docs: pull true-owner attribution out into its own explicit note

Was only stated as supporting detail inside the dedup-fix narrative -- someone looking specifically for "where does a shared reel end up?" had to read through the whole bug story to find it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011qAds5qr7nZRq5R4yAuxUk
feat: add reels-sync.sh, and dedupe reels-scrape.py against the whole archive
2026-08-27 17:52:39 -04:00 · 2026-08-27 14:54:39 -04:00 · 2026-08-27 14:41:49 -04:00 · 2026-08-27 14:36:09 -04:00 · 2026-08-26 20:45:46 -04:00 · 2026-08-26 20:43:38 -04:00
19 changed files with 3690 additions and 9 deletions
@@ -9,3 +9,5 @@ coverage/
 !.env.example
 _sample-archives
 _gemini-plans
 __pycache__/
 *.pyc
@@ -15,9 +15,6 @@ InstaArchive Viewer is a React 19 + Vite 6 PWA for browsing archived Instagram d
 - `npm run lint` — type-check only (`tsc --noEmit`)
 - `npm test` / `npm run test:watch` — vitest
 - `npx vitest run src/lib/archive-patterns.test.ts` — a single test file
 - `npm run jd2 -- --archives <dir> --dry-run` — generate JDownloader `.crawljob`
  files for every profile on disk (see `scripts/jd2-sync.ts` and
  `docs/jdownloader.md`)
 Local development usually needs both `npm run dev` and `npm run server`. Local-folder mode works without the backend; server-mode archives do not.
@@ -0,0 +1,751 @@
 # Tooling branch
 > [!CAUTION]
 > ## This branch must not be pushed to GitHub
 >
 > `tooling` is the only branch that still contains the archive-fetching
 > scripts and their docs, and those name things `main` was rewritten to
 > remove:
 >
 > - the fetch host's **public IP** (`docs/gallery-dl.md`)
 > - the browser profile the session cookie is read from
 > - the NAS archive path
 > - the **list of Instagram accounts being archived**
 >
 > On 2026-08-20 `main`'s entire history was rewritten with `git filter-repo`,
 > the GitHub repo was deleted and recreated, and 22 container images were
 > pruned from ghcr — all to get exactly this material out of public view.
 > **One push of this branch to GitHub undoes all of it.**
 >
 > A second cleanup would be harder than the first: after a force-push the old
 > commits stayed reachable by raw SHA, and only deleting the repository
 > outright removed them.
 ## Quick reference — getting new content onto the site
 Timers are **disabled**, so this is manual today. Everything runs on
 `mattellite`; it fetches, publishes to the NAS itself, and the site picks it up
 with no deploy or restart.
 ```sh
 ssh mattellite
 ~/gdl/gdl-cron.sh stories     # ~1 min.  DO THIS OFTEN - stories die in 24h
 ~/gdl/gdl-cron.sh profiles    # ~5 min.  posts, reels, highlights, stories
 ```
 `profiles` covers everything, so it is the one to run if you only run one — but
 it is no substitute for `stories`, because a story posted and expired between
 two `profiles` runs is simply gone. Both are safe to run back-to-back; the
 `--min-interval` floor makes a repeat a no-op rather than a re-fetch.
 Then check <https://instaarchive.ergosteur.com> — new posts appear without a
 redeploy. The archive index re-reads a directory when its mtime changes.
 **Reading the output.** These two look alarming and are fine:
 - `rsync error: ... (code 23)` / `done; 1 step(s) failed` — the `chown` to
  `rslsync` failing because the ssh user is not root. Files landed.
 - `No results` for a profile — it has no active story right now.
 **These mean stop**, and are covered in "When a run fails":
 - `429` from `scontent-*.cdninstagram.com`
 - `400 Bad Request` on `/api/v1/feed/reels_media/` — the account is behind a
  scraping-warning interstitial; check the browser before anything else.
 Long runs: `setsid nohup ~/gdl/gdl-cron.sh profiles >/dev/null 2>&1 &` and
 `tail -f ~/gdl/logs/$(ls -1t ~/gdl/logs | head -1)`. Nothing reaches the
 archive until a run finishes, so killing one midway is safe.
 ## Safety net: a global gallery-dl config
 `~/.config/gallery-dl/config.json` on `mattellite` exists so that a **plain
 `gallery-dl <url>` typed by hand** — for a quick manual check, outside
 `gdl-sync.py` entirely — still gets the hand-paced caution settings instead of
 gallery-dl's own faster defaults. It is loaded automatically; nothing needs to
 reference it. `gdl-sync.py`'s own `--sleep-request`/`--sleep`/`--rate` flags
 still override it as normal — this is only a floor for when nobody passed any.
 ```json
 {
    "extractor": {
        "instagram": {
            "api": "rest",
            "cookies": ["chrome", "/home/matt/.config/google-chrome-devtools"],
            "sleep-request": [12.0, 20.0],
            "sleep": [5.0, 10.0],
            "sleep-429": 120.0,
            "retries": 8,
            "videos": true
        }
    },
    "downloader": {
        "http": {
            "rate": "500K",
            "retries": 8
        }
    }
 }
 ```
 Every value here mirrors `gdl-cron.sh`'s own hand-paced defaults (see its
 `SLEEP_REQUEST`/`SLEEP`/`RATE` comments) and `build_config()` in
 `gdl-sync.py` — `api: rest` matters most: the graphql backend issues one
 request PER POST for every video and carousel, the pattern that got this
 account banned once already. `cookies` here is the config-file equivalent of
 `--cookies-from-browser`, so a bare `gallery-dl <url>` is already
 authenticated as the archive account, not anonymous.
 Verified with `gallery_dl.config.load()` + `config.get(...)` (zero live
 requests) — every value above loads correctly with no `--config` flag passed.
 This file is **not tracked in the repo** — like `artms.db` and the state
 files, it is host-local runtime config, and it embeds the same real
 Chrome-profile path already documented above. Recreate it by hand (or from
 this section) after a fresh `mattellite` setup.
 ## Remotes
 | remote | what goes there |
 |---|---|
 | `origin` → gitea | everything: `main`, `tooling`, tags, backups |
 | `github` | **`main` and the current release tag only** — it exists to run the CI/CD image build |
 The other 22 release tags stay on gitea. Pushing them all to GitHub triggers
 one container build per tag, because each tag carries its own workflow file.
 ## Guards — recreate these after a fresh clone
 Neither guard is versioned, so a new clone has **no protection at all**:
 ```sh
 git config remote.github.push refs/heads/main:refs/heads/main
 cat > .git/hooks/pre-push <<'HOOK'
 #!/bin/sh
 remote_url="$2"
 case "$remote_url" in *github.com*) ;; *) exit 0 ;; esac
 while read -r _ _ remote_ref _; do
  [ -z "$remote_ref" ] && continue
  case "$remote_ref" in
    refs/heads/main|refs/tags/*) ;;
    *) echo "pre-push: refusing to push '$remote_ref' to GitHub." >&2; exit 1 ;;
  esac
 done
 exit 0
 HOOK
 chmod +x .git/hooks/pre-push
 ```
 Decide on the **remote** ref, not the local one: a delete push sends
 `(delete)` as the local ref, and an earlier version of this hook rejected
 every deletion because of it.
 ## What lives here
 | path | what it is |
 |---|---|
 | `scripts/gdl-sync.py` | the gallery-dl fetcher; replaced JD2 for the ARTMS profiles |
 | `scripts/gdl-cron.sh` | unattended wrapper: `stories` \| `profiles` \| `full-sweep` |
 | `scripts/systemd/` | the timers actually installed on the fetch host |
 | `scripts/test_gdl_sync.py` | its tests |
 | `scripts/jd2-sync.ts` | JDownloader `.crawljob` generator, still used elsewhere |
 | `docs/gallery-dl.md` | the measurements behind every option in the fetcher — **read before changing pacing** |
 | `docs/jdownloader.md` | the older JD2 flow |
 | `docs/artms-instagram-accounts.txt` | the profile list passed to `--urls-file` |
 ## Commands
 ```sh
 # fetch: always --dry-run first; it prints the plan and the publish step
 ./scripts/gdl-sync.py --index https://instaarchive.ergosteur.com \
    --staging <dir> --publish <user>@<nas>:<archives> \
    --archive-db <db> --urls-file artms_account_links.txt --abort 50 --dry-run
 # crawljobs (no npm script — package.json is kept identical to main)
 npx tsx scripts/jd2-sync.ts --archives <dir> --dry-run
 ```
 Run the fetcher from the host whose public IP matches the browser the cookie
 came from. `--abort 50` is the routine setting; omit it for a full sweep that
 also catches edited carousels.
 ## Why there is no CLAUDE.md entry for any of this
 `CLAUDE.md`, `README.md` and `package.json` are kept **byte-identical** to
 `main` so that merging `main` into `tooling` never conflicts. The earlier
 attempt put tooling notes in `CLAUDE.md` and a `jd2` script in `package.json`;
 because `main` had *deleted* those lines, every merge re-applied the deletion.
 Keep branch-specific documentation in this file, which `main` does not have.
 ## Running it by hand
 Everything happens on **`mattellite`** — the fetch host whose public IP matches
 the browser the cookie came from. Running it anywhere else is what
 session-hijack detection looks for.
 ```sh
 ssh mattellite
 ~/gdl/gdl-cron.sh profiles    # or: stories | full-sweep
 ```
 That is the whole thing: it wipes staging, fetches, and publishes straight to
 the NAS. To drive `gdl-sync.py` directly instead — always `--dry-run` first,
 which prints the plan and the exact rsync that would touch the archive:
 ```sh
 cd ~/gdl
 PATH=$HOME/.local/bin:$PATH ./gdl-sync.py \
    --index    https://instaarchive.ergosteur.com \
    --staging  ~/gdl/staging-manual \
    --publish  agentapi@10.20.28.200:/volume1/rslsync/sync/Instagram-archive/archives/ \
    --archive-db ~/gdl/artms.db \
    --urls-file  ~/gdl/artms_account_links.txt \
    --sleep-request 12 20 --sleep 5 10 --rate 500K \
    --abort 50 --dry-run          # swap for --execute when the plan looks right
 ```
 **Do not omit the pacing flags.** `gdl-sync.py`'s own defaults are 6-10s / 3-6s
 / 1M, which is roughly half the caution the wrapper applies. A hand-run that
 leaves them off is *less* careful than the automation — which is backwards, and
 was true of this very example until 2026-08-22. Prefer `gdl-cron.sh`; it
 carries them for you.
 `PATH` matters: `gallery-dl` is a pipx install in `~/.local/bin`, which is not
 on cron's PATH and not on a non-login shell's either.
 Useful variations:
 ```sh
 --profile 0ct0ber19 --profile kimxxlip   # instead of --urls-file
 --only stories                            # one surface
 --max-sources 4                           # hard ceiling on what one run touches
 --force                                   # ignore --min-interval AND the probe
                                          # cache; almost never what you want
 ```
 ### Long runs
 A sync runs for minutes to hours at this pacing, so detach it rather than
 holding an ssh session open:
 ```sh
 cd ~/gdl && setsid nohup ./gdl-cron.sh profiles > /dev/null 2>&1 &
 tail -f ~/gdl/logs/$(ls -1t ~/gdl/logs | head -1)
 ```
 **Do not kill it with `pkill -f <pattern>` over ssh.** The pattern matches your
 own `ssh` command line, so you kill your own shell and the sync survives — this
 happened twice on 2026-08-22. Use `pkill -x chrome` style exact-name matches,
 or kill the pid: `pgrep -f 'only posts,reels' | tail -1`.
 Nothing reaches the archive until the run finishes: `gdl-sync.py` publishes
 once, at the end, so a run killed midway leaves the archive untouched.
 ### The three modes
 Renamed on 2026-08-22. `full` was misleading — it is the abort-*limited* run —
 and `sweep` did not convey that it is the exhaustive one. The wrapper rejects
 the old names with a pointer rather than a bare error.
 | mode | cadence | cost | why |
 |---|---|---|---|
 | `stories` | daily | ~6 requests | stories expire in 24h and **cannot be backfilled**; this is the only run that loses content if skipped |
 | `profiles` | monthly | ~40-60 requests | every surface, `--abort 50` — stops enumerating a profile once it reaches content already held. Catches everything **new** |
 | `full-sweep` | rarely, by hand | **~420 requests** | no abort; walks every profile to the end. The only run that notices posts **edited** after we archived them (test case 15) |
 The skip-archive means an infrequent `profiles` run costs barely more than a frequent
 one — it only fetches what is new. Frequency buys freshness, not completeness,
 except for stories.
 **Pacing is deliberately slower than `gdl-sync.py`'s own defaults.** All three
 modes run at `--sleep-request 12 20 --sleep 5 10 --rate 500K`, against defaults
 of 6-10 / 3-6 / 1M. These are the values the 2026-08-22 runs used by hand after
 the scraping warning, and they produced 0 400s and 0 429s. An archive sync has
 no deadline: being slow is free, being restricted is not. Override with
 `GDL_SLEEP_REQUEST`, `GDL_SLEEP`, `GDL_RATE` — to raise them, not lower them.
 `stories` also runs at `--min-interval 8` rather than the 20h default, because
 at 20h the daily timer silently did nothing whenever a manual run had happened
 the previous afternoon. The floor exists to stop an *aborted restart*
 re-enumerating profiles, which is a minutes-to-hours concern; a stories fetch
 is one request per profile, so 8h permits at worst about twelve requests in a
 day instead of six. **A stories run that skips every source now exits 75 and
 prints a warning** rather than reporting success.
 ## Scheduling — installed on `mattellite`
 systemd **user** timers, running as `matt`, with lingering enabled so they fire
 without a login session:
 ```sh
 loginctl show-user matt --property=Linger      # Linger=yes
 systemctl --user list-timers 'gdl-sync@*'
 ```
 | unit | schedule | next fire (as installed) |
 |---|---|---|
 | `gdl-sync@stories.timer` | daily 09:00 | 09:36:45 — the delay is the randomisation working |
 | `gdl-sync@profiles.timer` | 3rd of each month, 04:00 | 04:37:44 |
 | `gdl-sync@full-sweep.timer` | 7th of Jan/Apr/Jul/Oct, 04:00 | 04:42:39 |
 Unit files are version-controlled in `scripts/systemd/` and installed to
 `~/.config/systemd/user/`. One templated service, `gdl-sync@.service`, takes
 the mode as its instance name and runs `gdl-cron.sh %i`.
 Three settings are load-bearing:
 - **`RandomizedDelaySec=45m`** — a job firing at exactly 09:00 daily is
  obviously a machine, and the entire safety model is about not looking like
  one. This is why the table above shows 09:36 rather than 09:00.
 - **`Persistent=true`** — catch up a run missed because the host was off.
  cron silently skips, and a skipped `stories` run is content gone for good.
 - **`TimeoutStartSec=infinity`** — a sweep can run for hours at this pacing.
  The default 90s would kill it mid-fetch.
 Operating them:
 ```sh
 export XDG_RUNTIME_DIR=/run/user/$(id -u)      # needed over non-interactive ssh
 systemctl --user start   gdl-sync@stories.service   # run one now
 systemctl --user status  gdl-sync@profiles.timer
 journalctl --user -u 'gdl-sync@*' -n 50
 systemctl --user disable --now gdl-sync@full-sweep.timer   # stop one
 ```
 `systemctl --user` fails with "Failed to connect to bus" over ssh unless
 `XDG_RUNTIME_DIR` is set. Note also that **month names are not valid in
 `OnCalendar`'s date field** — `Jan,Apr,Jul,Oct-07` is rejected outright, hence
 `*-01,04,07,10-07`. Check any change with `systemd-analyze calendar '<expr>'`
 before installing it.
 ### cron, if you ever prefer it
 ```cron
 17 9  *  *  *        sleep $(shuf -i 0-2700 -n1); $HOME/gdl/gdl-cron.sh stories
 43 4  3  *  *        sleep $(shuf -i 0-2700 -n1); $HOME/gdl/gdl-cron.sh profiles
 11 4  7  1,4,7,10 *  sleep $(shuf -i 0-2700 -n1); $HOME/gdl/gdl-cron.sh full-sweep
 ```
 cron runs `/bin/sh`, so `$RANDOM` does not exist — hence `shuf`. And `%` in a
 crontab line means newline unless escaped, so avoid it entirely. cron has no
 equivalent of `Persistent=true`.
 ## Verifying a run
 The first unattended run is **2026-08-21, around 09:36** (09:00 plus the
 randomised delay). Nothing below costs an Instagram request — every check is
 against the journal, local logs, or our own viewer's API.
 ```sh
 # 1. did it run, and did it exit 0?
 ssh mattellite
 export XDG_RUNTIME_DIR=/run/user/$(id -u)
 systemctl --user list-timers 'gdl-sync@*'          # LAST/PASSED columns
 journalctl --user -u 'gdl-sync@stories.service' --since yesterday --no-pager
 # 2. what did it actually fetch? (sidecars vastly outnumber media -- count media)
 ls -1t ~/gdl/logs | head -3
 grep -E '^(==>|    FAILED|done;)' ~/gdl/logs/stories-*.log | tail -20
 grep -ic 429 ~/gdl/logs/stories-*.log            # MUST be 0 -- see below
 ```
 **A `429` from `scontent-*.cdninstagram.com` ends the session, it is not a
 pacing knob to tune.** The warning order last time was CDN 429 → `400` on the
 highlights endpoint → suspension. If a run logs one, disable the timers and
 stop for the day:
 ```sh
 systemctl --user disable --now gdl-sync@stories.timer gdl-sync@profiles.timer gdl-sync@full-sweep.timer
 ```
 Then confirm the archive actually grew, from the workstation:
 ```sh
 # 3. did the publish land? compare against yesterday's counts
 for u in 0ct0ber19 kimxxlip withaseul cher_ryppo zindoriyam official_artms; do
  n=$(curl -s "https://instaarchive.ergosteur.com/api/archives/$u/files" | python3 -c 'import json,sys; d=json.load(sys.stdin); print(len(d if isinstance(d,list) else d["files"]))')
  printf '%-18s %s\n' "$u" "$n"
 done
 ```
 Counts after the 2026-08-22 runs, to diff against:
 | profile | files |
 |---|---:|
 | 0ct0ber19 | 3160 |
 | official_artms | 6760 |
 | cher_ryppo | 3056 |
 | kimxxlip | 3021 |
 | zindoriyam | 2256 |
 | withaseul | 1760 |
 A stories-only run adds few files and often **none** — profiles frequently have
 no active story. "0 new" is a normal result, not a failure. `fileCount` in
 `/api/archives` is stale by design; use the per-profile `/files` listing.
 ## 2026-08-21 — scraping warning, automation stopped
 **Instagram flagged the account.** Not suspended: an interstitial at
 `/accounts/scraping_warning/` reading *"We suspect automated behaviour on your
 account"*. It was dismissed in the browser and the account is healthy — feed
 loads, still signed in. **All three timers are disabled.** Do not re-enable
 them without deciding the cadence question below.
 How it unfolded, because each step misled in a different way:
 1. **09:13** the daily timer fired, exited 0 in one second, logged
   `done; 0 step(s) failed` — and fetched nothing. Yesterday's manual run was
   18.9–19.2h earlier, just under the `--min-interval 20` floor, so all six
   sources were skipped. **A silent no-op on the one surface that cannot be
   backfilled, reported as success.**
 2. **20:28** `chrome-devtools.service` was OOM-killed (5.1 GB peak, ~1w3d CPU).
   Unrelated to the above, and it does not break fetching — gallery-dl reads
   the cookie *file*, not a live browser — but it meant no browser was running
   to notice anything was wrong.
 3. **23:19** a manual recovery run passed the floor (33h) and every source
   failed with `400 Bad Request` on
   `/api/v1/feed/reels_media/?reel_ids=…`. Six identical failures across six
   profiles is not a per-profile fault.
 4. Cookies were **exported and checked before assuming a block**:
   `sessionid` 77 chars, printable, colon-delimited, 360 days to expiry;
   `ds_user_id` present. Decryption was fine, so the fault was server-side.
   This check costs no Instagram requests and should always come first.
 5. The browser then showed the interstitial. The 400s were the challenge
   state, not a ban.
 ### What has to change before automation is re-enabled
 - **The `--min-interval` floor silently defeats the daily job.** Any manual run
  in the preceding 20h makes the scheduled one a no-op. The floor exists to
  stop an *aborted restart* re-enumerating profiles — a minutes-to-hours
  concern — and a stories fetch is one request per profile. `stories` should
  use something like `--min-interval 8`, not 20.
 - **A skipped stories run must be loud.** `0 to sync, 6 skipped` currently
  exits 0 and looks identical to success. On this surface a skip is a real
  loss, and it should be visible in the journal without reading the log.
 - **Reconsider the daily cadence itself.** A job hitting story endpoints for
  six profiles every morning is the most machine-like thing here, randomised
  delay or not, and it is what was flagged. Every-few-days, or on-demand, may
  be the honest answer even though stories will be missed.
 - **`chrome-devtools.service` has `Restart=no`** and died silently for three
  hours. It needs `Restart=on-failure` and probably a `MemoryMax=`, or it will
  be dead the next time the cookie needs refreshing.
 ### 2026-08-22 — caught up by hand, cleanly
 Both surfaces were fetched manually the next day, on the owner's call, at
 **roughly double the configured caution**: `--sleep-request 12 20`,
 `--sleep 5 10`, `--rate 500K`, versus the defaults of 6-10 / 3-6 / 1M.
 | run | result |
 |---|---|
 | stories, 6 profiles | 16 media, +21 files, **0 400s, 0 429s** |
 | posts+reels, 12 sources, `--abort 50` | 26 media, +64 files, **0 400s, 0 429s** |
 So the 400s really were the challenge state and nothing more: once the
 interstitial was dismissed in the browser, the same endpoints served normally.
 The stories that looked lost — `official_artms`, `0ct0ber19`, `cher_ryppo`,
 `kimxxlip`, `zindoriyam` — were all captured before expiry.
 **This does not retire the warning.** Two hand-paced runs a day later are not
 evidence that the previous cadence was safe; they are evidence that the
 account still works. What actually changed the request cost is `--abort 50`:
 twelve sources across six profiles, `official_artms` included at 1829 posts
 and 781 reels, finished in minutes for a few dozen requests where the old
 behaviour would have spent ~400.
 Note the gap this leaves: **the scheduled `profiles` mode still uses the default
 6-10s pacing**, not the 12-20s used here. Reconcile that before re-enabling
 the timers, or the automation will be less careful than the hand runs that
 followed a warning.
 ## When a run fails
 Worked through on 2026-08-22. Do these **in order** — the first two cost no
 Instagram requests, and the third costs one page view.
 ### 1. Read the error, not the exit code
 ```sh
 L=~/gdl/logs/$(ls -1t ~/gdl/logs | head -1); grep -vE '^(profiles|surfaces|sources|pacing|staging|publish|read )' "$L" | head -40
 ```
 Two results are **not** failures, despite how they look:
 | looks like | actually |
 |---|---|
 | `rsync error: some files/attrs were not transferred (code 23)` and `done; 1 step(s) failed` | the `chown` to `rslsync` failing because the ssh user is not root. Data landed. Confirm by re-running the rsync with `--dry-run`: an empty file list means everything arrived |
 | a source reporting `No results` | that profile simply has no active story / no highlights |
 ### 2. Check the cookies before concluding you are blocked
 Free, and it separates a local fault from a server-side one:
 ```sh
 PATH=$HOME/.local/bin:$PATH gallery-dl \
  --cookies-from-browser "chrome:/home/matt/.config/google-chrome-devtools" \
  --cookies-export /tmp/ck.txt
 grep instagram /tmp/ck.txt | awk '{print $6, length($7)}'   # names + value lengths
 rm -f /tmp/ck.txt
 ```
 A healthy `sessionid` is ~77 chars, printable ASCII, colon-delimited, and
 unexpired; `ds_user_id` should be an 11-digit number. Garbage or non-printable
 values mean Chrome's cookie decryption failed locally — not that the account is
 in trouble. **Never print the values into a transcript or a bug report.**
 ### 3. Look at the account in the browser
 The session lives in a dedicated Chrome on `mattellite`, on display `:1` with
 CDP on 9222, run by a systemd unit:
 ```sh
 export XDG_RUNTIME_DIR=/run/user/$(id -u)
 systemctl --user status chrome-devtools.service
 systemctl --user start  chrome-devtools.service    # it has Restart=no
 ```
 Then VNC to `:1` and open `instagram.com`. What you are looking for:
 - **`/accounts/scraping_warning/`** — *"We suspect automated behaviour on your
  account"*. This is what `400 Bad Request` on `/api/v1/feed/reels_media/`
  actually means; the session is in a challenge state, not banned. Dismissing
  it in the browser restores API access immediately, verified 2026-08-22.
 - a checkpoint or login page — the session is gone; re-log in the browser.
 - a normal feed — the fault is elsewhere.
 **Do not dismiss a warning and immediately resume.** The dismissal fixes the
 symptom. The behaviour that caused it is the thing to change.
 ### 4. Stop, if it was a 429 or a repeated 400
 ```sh
 systemctl --user disable --now gdl-sync@stories.timer gdl-sync@profiles.timer gdl-sync@full-sweep.timer
 ```
 The documented escalation is CDN 429 → 400 on a stories/highlights endpoint →
 suspension. It has now run twice, and both times the 400 was the last warning
 before something worse.
 ## Reels: the API is blocked, scrape by scrolling instead
 As of 2026-08-26/27, `gallery-dl`'s dedicated reels extractor fails on every
 profile with `HTTP redirect to home page` — confirmed hours apart, with a
 freshly-warmed session and a correct `X-IG-WWW-Claim` header (that was the
 first suspect; ruled out by tracing the raw HTTP exchange). It is not a
 scraping-warning interstitial — the reels tab loads completely normally in a
 real, already-signed-in browser — so this is Meta blocking the specific
 `/api/v1/clips/user/` endpoint gallery-dl calls, not an account-health issue.
 Posts, highlights and stories are unaffected; only reels breaks.
 `scripts/reels-scrape.py` works around it by never calling that endpoint:
 it drives the same signed-in Chrome via its loopback CDP port (`:9222`,
 already exposed for MCP automation), scrolls the reels tab like a person
 would, and scrapes `/reel/<code>/` links out of the rendered page. It only
 finds shortcodes — nothing is downloaded until the fetch step — and dedupes
 against the **whole archive**, not just the profile being scraped (see why
 below), so re-running it costs nothing for reels already held anywhere.
 ### The easy way: `reels-sync.sh`
 ```sh
 ssh mattellite
 ~/gdl/reels-sync.sh zindoriyam
 # or a full URL: ~/gdl/reels-sync.sh https://www.instagram.com/zindoriyam/
 ```
 Runs both steps (scrape, then fetch + publish whatever's new) with the same
 hand-paced settings `gdl-cron.sh` uses, logs to `~/gdl/logs/reels-<profile>-*`,
 and exits 0 with "no new reels" printed when a profile is already caught up
 — `gdl-sync.py` never even gets invoked in that case. Override pacing the
 same way as `gdl-cron.sh` (`GDL_SLEEP_REQUEST`, `GDL_SLEEP`, `GDL_RATE`), plus
 `GDL_SCROLL_PAUSE` and `GDL_MAX_IDLE_ROUNDS` for the scrape step. Staging
 (`~/gdl/staging-reels-<profile>`) and the scraped URL list
 (`~/gdl/<profile>-reels.txt`) are wiped at the START of the next run, not
 after — left behind for inspection, same as `gdl-cron.sh`'s `staging-*`.
 Verified end to end against `zindoriyam` on 2026-08-27: 26 reels found on the
 page, 10 new ones fetched and published cleanly on the first run.
 **Not wired into `gdl-cron.sh` or the timers.** Both scripts drive your
 actual browser session rather than firing a background request, and are
 slower by design (real scrolling, not an API call) — both good reasons not
 to run this unattended without deciding that deliberately. Today it is a
 per-profile, by-hand tool only.
 ### Where a shared reel lands
 A reel found on one profile's `/reels/` page is not necessarily *owned* by
 that profile — reposts and collabs between tracked accounts show up there
 too. It always gets filed under its **true owner**, per Instagram's own
 metadata on the post, never under whichever profile's page you happened to
 scrape it from — `gdl-sync.py`'s directory template for a scraped reel is
 `{username}` filled in from that metadata, the same mechanism highlights
 already used for their own directory. So:
 - Scrape order doesn't matter. Run `reels-sync.sh` on `0ct0ber19` or
  `zindoriyam` first, whichever — a reel they share lands in the same place
  either way, and running it on the other one afterward just sees that
  shortcode as already archived (dedup checks every profile, not only the
  one being scraped) and skips it.
 - It never gets duplicated into both accounts' directories, and it never
  gets misattributed to the profile you scraped instead of who actually
  posted it.
 This is also literally why the whole-archive dedup fix above exists: a
 zindoriyam-scraped shortcode that turned out to belong to `0ct0ber19` was
 filed under `0ct0ber19/`, not `zindoriyam/` — dedup that only checked
 `zindoriyam`'s own directory would never have found it there.
 ### What it's doing, if you want to run the two steps separately
 ```sh
 PROFILE=someuser
 # 1. Scrape by scrolling; dedupe against the archive; write new URLs to a file.
 #    Needs gallery-dl's own pipx venv python -- that's where websocket-client
 #    (the one extra dependency this needs) got injected.
 ~/.local/share/pipx/venvs/gallery-dl/bin/python3 ~/gdl/reels-scrape.py \
  --profile "$PROFILE" \
  --index https://instaarchive.ergosteur.com \
  --out ~/gdl/"$PROFILE"-reels.txt
 # 2. Fetch whatever's new -- dry run first, same as any other gdl-sync.py call.
 cd ~/gdl && PATH=$HOME/.local/bin:$PATH ./gdl-sync.py \
  --publish agentapi@10.20.28.200:/volume1/rslsync/sync/Instagram-archive/archives/ \
  --staging ~/gdl/staging-reels-scraped \
  --sleep-request 12 20 --sleep 5 10 --rate 500K \
  --post-urls-file ~/gdl/"$PROFILE"-reels.txt --dry-run
 # ...then swap --dry-run for --execute once the plan looks right.
 ```
 Notes:
 - `reels-scrape.py` fails fast if Chrome/CDP isn't up
  (`curl -s http://localhost:9222/json/version` to check first).
 - If it finds nothing new, `gdl-sync.py --post-urls-file` refuses to run
  ("no usable URLs") rather than doing nothing quietly — expected when a
  profile is already caught up.
 - `--max-idle-rounds` (default 3) and `--scroll-pause MIN MAX` (default
  `2.0 3.5`) are worth raising for an unusually large or slow-loading reels
  tab; the default stops once 3 consecutive scrolls find nothing new.
 ### Why dedup checks the whole archive, not just the scraped profile
 First version deduped only against the scraped profile's own directories.
 Re-running it on `zindoriyam` minutes after a successful fetch found "9 new"
 reels again — all reposts/collabs originally by `0ct0ber19` and
 `official_artms` (also tracked profiles). The *old*, broken direct-reels-tab
 fetch had left orphaned sidecar-only remnants (`.json`/`.txt`, no media) for
 them, misfiled under `zindoriyam - reels/` with the true owner's name baked
 into the filename stem — `index_existing()` correctly does not count a
 sidecar-only entry as "held", so they looked new. `gdl-sync.py` re-fetched
 them, filed them correctly under `{username}` (the post's *true* owner, per
 its own metadata) — where they already existed from that profile's own
 regular sync — and `rsync --ignore-existing` silently skipped every one, so
 nothing was lost or duplicated. But 9 Instagram requests were spent finding
 that out. Since a shortcode is globally unique, `reels-scrape.py` now checks
 every archived profile's listing, not just the one being scraped — all local
 requests to the viewer's own API, never to `instagram.com`, so checking all
 16 profiles costs nothing on the budget that actually matters. Confirmed
 fixed the same day: re-running against `zindoriyam` immediately afterward
 found "26 already archived, 0 new" and exited clean.
 ## What changed on 2026-08-20
 One session, three separate pieces of work. Recorded because the reasons are
 not recoverable from the diffs.
 **The sync run.** First incremental fetch in four days: 184 new media, 299
 files published, 0 failures, 0 CDN 429s. 20 story items, which are the part
 that could not have been recovered later. Cost about half what it would have,
 because the archive DB was already seeded and the state file was primed by hand
 so no probe passes ran.
 **`--abort 50`.** The skip-archive suppresses *downloads*, which spends the
 CDN; it does nothing about the *listing pass*, which spends `instagram.com` and
 scales with how big a profile is rather than how much is new. Measured from
 sidecar write times mid-run: 3 new posts took ~100s each, the other 2272 were
 written in one second. Enumerating `cher_ryppo` fell from 2151 posts to 7.
 **The repo split.** `main` is public and now carries none of the fetching
 tooling, no host details, and no real account names — its entire history was
 rewritten, the GitHub repo deleted and recreated to clear force-push residue,
 and 22 container images pruned from ghcr because the server bundle had been
 shipping source comments naming real accounts. This branch holds everything
 that was removed. See the caution at the top.
 **Automation.** mattellite got a key on the NAS, closing the last manual step,
 and three systemd timers now run the sync unattended.
 ## Outstanding
 State as of 2026-08-20, after the sync run and the repo split. Nothing here is
 broken; these are decisions not yet made and cleanups not yet done.
 ### Fetching
 - **The timers are DISABLED** after the 2026-08-21 scraping warning; see that
  section. Their one unattended firing did the wrong thing — it skipped every
  source on the 20h floor and reported success — so re-enabling should wait on
  the four fixes listed there, not just on the account settling.
 - `mattellite`'s `~/.ssh/id_ed25519.pub` is in the NAS's `authorized_keys` for
  `agentapi` (added 2026-08-20, alongside the workstation's existing key), so
  the fetch host publishes straight to the archive and no `sshpass` step is
  needed. **That key is what makes the timers work** — remove it and every
  scheduled run will fetch successfully and then fail at publish.
 - **5.4 GB of stale staging on `mattellite`** (`~/gdl`, 44 GB free): `staging`
  and `out` at 2.2 GB each from 2026-08-17, `staging-0820` / `out-0820` at
  446 MB each, plus `staging-full` (78 MB) and `staging-stories` (66 MB) from
  2026-08-22. All of it was verified published, so all of it is safe to delete.
  Staging is wiped per-run by `gdl-cron.sh`, but the `out-*` publish targets and
  anything created by a direct `gdl-sync.py` call are never cleaned up.
 - **Stories currently depend on someone remembering.** The daily timer exists
  but is disabled, so the one surface that cannot be backfilled has no
  automation behind it. Every day nobody runs `gdl-cron.sh stories` is a day
  of stories gone. That is the central unresolved tension: the cadence that
  protects stories is also the most machine-like pattern here.
 - ~~The scheduled modes have not been reconciled with the pacing used by
  hand.~~ **Done 2026-08-22**: all modes now pass `--sleep-request 12 20
  --sleep 5 10 --rate 500K`, `stories` uses `--min-interval 8`, and a
  fully-skipped stories run exits 75 with a warning instead of looking like a
  success. The timers are still **disabled** — enabling them is a separate
  decision about cadence, not about pacing.
 - **`--abort 50` is opt-in.** `gdl-cron.sh profiles` passes it and the manual
  runs used it; `full-sweep` deliberately does not. It stops noticing **edited
  carousels** (test case 15), which only a full enumeration finds — which is
  what `full-sweep` is for. No cadence for it was ever agreed, and at ~420
  requests it is the riskiest thing on the schedule; prefer running it by hand.
 - **The `seeded` flags in `<db>.state.json` were hand-written**, reconstructed
  from the 2026-08-17 log rather than derived from the archive DB. They assert
  "the skip-archive already knows this source". If `artms.db` is ever rebuilt,
  moved or lost, **clear the state file too** — otherwise those sources will
  never re-seed and a fetch into empty staging re-downloads everything.
 - **`~/gdl/gdl-sync.py` on the fetch host is a copy, not a checkout.** It
  currently matches this branch (`96e5694e…`), but nothing keeps them in sync;
  `scp` it after any change and re-check the hash.
 - The 2026-08-20 run is split across two logs — `artms-run3.log` (12 sources,
  no abort) and `artms-run4.log` (12 sources, `--abort 50`) — because it was
  stopped midway to pick up the new flag.
 ### Repo and infrastructure
 - **The `pre-rewrite-*` branches on gitea hold the unredacted history** — real
  account names, the fetch host's IP, and the tooling, as it was before the
  rewrite. They are deliberate backups. Decide whether they expire; the
  `pre-push` hook does cover them (it allows only `main` and tags to GitHub).
 - **The `pre-rewrite-full.bundle` backup is in a session scratchpad** and will
  be deleted with it. If a durable backup outside gitea is wanted, move it now.
 - **Only one container image exists.** 22 versions were pruned, so rolling back
  to an older release means checking out its tag from gitea and pushing that
  tag to GitHub to rebuild it — the old images are gone, not archived.
 - **CI warns that the Node 20 actions are deprecated.** `actions/checkout@v4`,
  `docker/login-action@v3`, `docker/metadata-action@v5` and
  `docker/build-push-action@v5` are being forced onto Node 24. They work today;
  bump when convenient.
 - **`review-fixes`** on gitea is a stale v1.3.0-era branch, never merged,
  published only because the whole local repo was pushed. Probably deletable.
 - GitHub Actions run history was lost when the repo was recreated. Cosmetic.
@@ -0,0 +1,6 @@
 https://www.instagram.com/0ct0ber19/
 https://www.instagram.com/kimxxlip/
 https://www.instagram.com/withaseul/
 https://www.instagram.com/cher_ryppo/
 https://www.instagram.com/zindoriyam/
 https://www.instagram.com/official_artms/
@@ -0,0 +1,610 @@
 # gallery-dl — a CLI replacement for JDownloader2
 Status: **in production.** All six ARTMS profiles are synced with
 `scripts/gdl-sync.py`; JD2 is no longer used for them.
 Everything below was measured against the live site and the real archive on
 2026-08-16 and 2026-08-20, not inferred from documentation.
 ## Why gallery-dl and not a hand-rolled script
 The hard parts of fetching Instagram are pagination, cookie handling, CDN URL
 expiry and resumption. gallery-dl already has all of them, plus extractors that
 map 1:1 onto our sidecar directory layout (`posts`, `reels`, `stories`,
 `highlights`). Rolling our own would mean reimplementing the ban-sensitive part
 by hand.
 ## The account was suspended on 2026-08-17 — read this first
 The account used for all of the below was suspended the same day this tooling
 was built, for "activity that doesn't follow our Community Standards on spam".
 The fetching was not the expensive part. **Verification was.**
 **It was restored, and synced normally again on 2026-08-20** — a full run
 across all six profiles with 0 failures and 0 CDN 429s. That is not evidence
 the limits were imagined; it is one data point on a restored account that has
 been treated carefully since. Everything below still applies, and the budget is
 still per session rather than per command.
 What was actually spent against `instagram.com` in a few hours, from one
 session and one IP:
 | activity | rough requests | downloaded |
 |---|---:|---|
 | enumerating a profile grid by scrolling it in an automated browser | ~18 pages | nothing |
 | the same profile again, after a bug in the scraping selector | ~18 pages | nothing |
 | a Reels tab enumerated the same way | ~9 pages | nothing |
 | full `-j` metadata dumps of one profile, twice | ~16 pages | nothing |
 | `--simulate` runs over the same profile, three times | ~24 pages | nothing |
 | single-post `/p/<code>/` fetches while testing filename formats | ~8 | a handful |
 | an aborted sync that re-ran every listing pass before dying | ~40 pages | ~270 MB |
 | the real sync, 24 sources across 6 profiles | ~150 pages | 2.2 GB |
 The two rows that actually mattered to the archive are the last one and part of
 the second-to-last. **Everything above them produced no files at all**, and
 together they were a comparable number of requests.
 The warnings arrived in this order and were each rationalised:
 1. `429 Too Many Requests` from `scontent-*.cdninstagram.com`, losing two
   videos. Treated as a pacing problem — pacing was lowered and the run
   continued.
 2. `400 Bad Request` from `/api/v1/highlights/<id>/highlights_tray/`, on an
   endpoint that had worked hours earlier. Correctly read as a possible block;
   requests stopped.
 3. Suspension.
 **Treat the first CDN 429 as a stop signal for the session, not a tuning
 parameter.** It is the tolerant surface complaining; if that surface is
 complaining, the rate-limited one has been unhappy for a while.
 ### Rules that follow from this
 - **Count verification requests against the same budget as fetching.** A
  `--simulate`, a `-j` dump and a browser scroll all hit `instagram.com` and
  download nothing. Being read-only does not make them free; it makes them
  invisible, which is worse.
 - **Never enumerate the live site with an automated browser.** Scrolling a
  214-post grid is ~18 paginated GraphQL loads at machine speed with no dwell
  time between them. It is the most obviously non-human thing in this whole
  document, and it was done here twice on one profile.
 - **Verify against the archive, not against Instagram.** Every naming, dating
  and classification question answered in this file could have been answered
  from files already on disk plus a single listing pass.
 - **`probe_live` is not cached, so every restart re-enumerates everything.**
  The aborted run cost a full duplicate set of listing passes for five
  profiles. Cache probe output to disk before running anything twice.
 - **Budget per session, not per command.** Nothing in the tooling knows what
  the last command spent.
 ### For a replacement account
 - Let it exist and be used normally for a while before pointing any tool at it.
 - Keep the cookie on one machine and one public IP, as before.
 - Start with a single small profile and stop for the day afterwards.
 - Prefer Instagram's own "Download a copy" export where possible: it is
  first-party, costs no scraping requests, and carries the metadata this whole
  document works around not having.
 ## The safety model — read this before changing any option
 The ban vector is **requests to `instagram.com`**, not bandwidth. See
 `docs/jdownloader.md` for the history; Instaloader got this account banned by
 asking `instagram.com` a question *per post*.
 gallery-dl has two API backends and the difference is exactly that vector:
 ```python
 if self.config("api") == "graphql":
    self.api = InstagramGraphqlAPI(self)   # per-post api.media() for every
 else:                                      # video and every carousel
    self.api = InstagramRestAPI(self)      # <- default, listing-only
 ```
 The REST backend paginates at `count: 30` (feed) / `page_size: 50` (clips), and
 those responses already carry `carousel_media`, `image_versions2`,
 `video_versions` and `product_type`. **No per-post request.** A 300-post
 profile costs roughly 10 requests to `instagram.com`.
 Rules, in order of importance:
 1. **`"api": "rest"` always.** Never `graphql`. This is the whole ballgame.
 2. **Never enable `metadata`-style options that trigger extra calls.** If a
   field is not already in the listing response, it is not worth a request.
 3. **Pace it.** `"sleep-request": [4.0, 7.0]` — a randomised gap, not a fixed
   one. Also `"sleep": [1.0, 3.0]` between downloads.
 4. **Cap the download rate** (`downloader.http.rate`) so the CDN side looks like
   a person, not a mirror.
 5. **Run from the same public IP as the browser the cookie came from.** At time
   of writing that is `mattellite` (`66.23.52.196`); the dev workstation is a
   *different* public IP and using the cookie from there is precisely what
   session-hijack detection looks for.
 6. **No programmatic login, ever.** gallery-dl's username/password path is
   disabled upstream anyway; use `--cookies-from-browser`.
 Do not add proxy rotation, fingerprint spoofing or account rotation. Throttling
 and request-avoidance are welcome; evasion is not.
 ### Cookies
 The logged-in Chrome on `mattellite` runs with a non-default profile:
 ```
 --user-data-dir=/home/matt/.config/google-chrome-devtools
 ```
 so the cookie flag is:
 ```
 --cookies-from-browser "chrome:/home/matt/.config/google-chrome-devtools"
 ```
 Plain `--cookies-from-browser chrome` fails with "Unable to find chrome cookies
 database" because it looks in `~/.config/google-chrome/`.
 Anonymous access is **not** a viable fallback: it serves lower-resolution media,
 caps profile pagination at 12 posts, and returns `AuthRequired` for stories and
 highlights.
 ## Output format
 The viewer's parser is the contract, not JD2's exact bytes. `EXPORT_RE` in
 `src/lib/archive-patterns.ts` accepts all of these, and normalises the index
 with `parseInt`, so **JD2 and gallery-dl naming interoperate**:
 ```
 "… - CrORBIcJJbM.mp4"      -> postId=CrORBIcJJbM  index=1
 "… - CrORBIcJJbM - 1.mp4"  -> postId=CrORBIcJJbM  index=1
 "… - C53YPQzp7Wj - 09.jpg" -> postId=C53YPQzp7Wj  index=9
 ```
 That means zero-padding and the presence/absence of ` - N` on single-media posts
 are cosmetic. Don't spend effort forcing them.
 ### Directory layout
 | kind | directory | note |
 |---|---|---|
 | posts | `<user>` | |
 | reels | `<user> - reels` | |
 | stories | `story - <user>` | |
 | highlights | `story highlights - <user> - <title>` | |
 **Force the directory with `-D`; never use `{username}` for it.** A profile's
 reels tab returns *collab reels owned by other accounts* — `/0ct0ber19/reels/`
 served 6 reels owned by `official_artms` and 1 by `chuuo3o`. With
 `{username}` those would scatter into `official_artms - reels/`. JD2 got this
 right and the archive proves it: `chuuo3o` and `official_artms` filenames sit
 inside `0ct0ber19 - reels/`.
 So: **owner in the filename, crawl scope in the directory.**
 ### Filenames
 ```jsonc
 "filename": {
  "sidecar_shortcode and count >= 10":
      "{date:Olocal/%Y-%m-%d}_{username} - {post_shortcode} - {num:02}.{extension}",
  "sidecar_shortcode":
      "{date:Olocal/%Y-%m-%d}_{username} - {post_shortcode} - {num}.{extension}",
  "":
      "{date:Olocal/%Y-%m-%d}_{username} - {post_shortcode}.{extension}"
 }
 ```
 `sidecar_shortcode` is set only when the post is a carousel, so it is the
 carousel discriminator. Conditions are evaluated in order, first match wins
 (`path.py:265`).
 Stories and highlights use the per-item `{shortcode}`, not `{post_shortcode}`
 (which is the *reel's* id, shared by every item in it):
 ```
 "{date:Olocal/%Y-%m-%d}_{username} - {shortcode}.{extension}"
 ```
 `{date}` on a story/highlight file is the **per-item** `taken_at`
 (`instagram.py:337` prefers `item["taken_at"]`), verified on a 154-item
 highlight whose items carried distinct times while `post_date` stayed pinned to
 the reel. Highlights therefore gain real dates — today they fall back to
 directory mtime.
 ### The timezone is not UTC
 JD2 stamped filenames in **desktop local time (US Eastern)**. Measured across
 212 comparable posts:
 | model | mismatches |
 |---|---:|
 | UTC | 19 |
 | UTC−5 (EST) | 10 |
 | UTC−4 (EDT) | **0** |
 | America/New_York (DST-aware) | **0** |
 `{date:Olocal/%Y-%m-%d}` uses the machine's local zone with per-timestamp DST
 awareness, which reproduces it — `mattellite` is `America/Toronto`, the same
 offsets. Note the **trailing `/` must be omitted**: `Olocal/%Y-%m-%d/` puts the
 separator into the strftime format and it sanitises to an underscore, giving
 `2026-08-15__0ct0ber19`.
 If the sync ever moves to a host in another timezone, set an explicit
 `{date:O-4/…}` or the dates will silently shift for ~9% of posts.
 ### Caption sidecars
 JD2 writes one `.txt` per post, named without the index, containing the caption
 with **no trailing newline**, and writes nothing when the caption is empty
 (measured: 197 of 217 posts, 86 of 86 reels, 0 of 10 stories, 0 of 16
 highlights). gallery-dl reproduces this exactly with the default
 `"empty": false`:
 ```jsonc
 { "name": "metadata", "event": "post", "mode": "custom",
  "content-format": "{description}", "extension": "txt",
  "filename": "{date:Olocal/%Y-%m-%d}_{username} - {post_shortcode}.txt" }
 ```
 `"event": "post"` is what makes it one file per post rather than per media file.
 ### Metadata sidecar (new — JD2 had no equivalent)
 ```jsonc
 { "name": "metadata", "event": "post", "mode": "json",
  "filename": "{date:Olocal/%Y-%m-%d}_{username} - {post_shortcode}.json",
  "include": ["post_shortcode","post_id","type","date","post_date","username",
              "fullname","owner_id","description","count","likes","post_url",
              "sidecar_shortcode"] }
 ```
 Use **`include`**, not `fields` — `fields` is for `mode: custom` and silently
 does nothing here, leaving `audio_user` blobs (including another user's profile
 picture URL) in the output.
 The payoff is `type`, which is Instagram's own classification:
 ```json
 { "post_shortcode": "DbdG9L9jU4m", "type": "post",  "count": 2 }   // feed video
 { "post_shortcode": "Db-lNCoib9m", "type": "reel",  "count": 1 }   // real reel
 ```
 This is the `product_type: "clips"` signal, delivered free in the listing
 response. It is the authoritative answer to "is this a reel", and would let the
 viewer retire the lone-video heuristic in `src/lib/post-tabs.ts` — see
 "Scanner work" below.
 **`type` is only populated by listing extractors.** Extracting a single
 `/p/<shortcode>/` URL leaves it `null`. Sync always uses listing URLs, so this
 only matters when testing by hand.
 ## Cadence, and the budget that enforces it
 **Monthly for everything, daily for stories only.** Stories expire in 24h and
 cannot be backfilled, so they are the one surface where missing a day means
 losing the content permanently. Everything else can wait — the skip-archive
 means an infrequent full sync costs barely more than a frequent one, because it
 only fetches what is new.
 ```
 # monthly, everything
 gdl-sync.py --index <viewer-url> --staging ~/gdl/staging \
    --publish <user>@<nas>:<archives> --archive-db ~/gdl/artms.db \
    --urls-file artms_account_links.txt --execute
 # daily, stories only -- one request per profile
 gdl-sync.py ... --only stories --execute
 ```
 A stories-only run is one source per profile and **never seeds**, because a
 story cannot be in the archive before it is fetched; probing would double the
 cost of the cheapest surface for no benefit. Six profiles is a handful of
 requests.
 When scheduling it, **randomise the minute and avoid the hour boundary**. A job
 that fires at exactly 09:00 every day is a machine; one that fires somewhere in
 a window looks like someone opening the app.
 The tool now refuses to repeat itself:
 | flag | default | what it prevents |
 |---|---|---|
 | `--min-interval` | 20h | re-fetching a source touched recently — the aborted-restart case that re-enumerated five profiles |
 | `--probe-ttl` | 24h | paying for a listing pass twice within a run cycle |
 | `--max-sources` | off | a runaway list touching more than intended |
 | `--force` | off | (escape hatch: ignores both guards) |
 State lives beside the archive DB as `<db>.state.json`, recording per source
 when it was seeded and last fetched. **Seeding is a one-time bootstrap**: after
 the first successful sync the archive DB records everything gallery-dl has
 seen, so the source is never probed again. That is the single biggest saving
 here — a second full sync costs roughly half what the first did.
 ## Incremental sync — why the fetch host needs no copy of the archive
 gallery-dl can skip already-held media two ways, and the difference decides
 whether the fetcher needs the archive mounted:
 - **By file existence** (default). Needs the destination to already contain the
  files, so it only works if the archive is mounted where gallery-dl writes.
 - **By skip-archive** (`--download-archive`). A sqlite DB of ids. Needs nothing
  on disk.
 We use the second, so the fetch host can write to **local disk and rsync
 afterwards**. That avoids writing tens of thousands of small files over CIFS,
 and keeps a mid-sync failure from leaving partial files on the live Resilio
 share.
 The key is `archive_prefix + archive_fmt`, which for this extractor is the
 literal `instagram` plus the per-media numeric pk (`instagram.py:25`,
 `job.py:713-719`). Verified: a 3-image carousel produced
 ```
 instagram3079387627521318672
 instagram3079387627521429433
 instagram3079387627529716672
 ```
 and a second run skipped every media file, rewriting only the idempotent
 `.txt`/`.json` sidecars.
 **Seeding.** `media_id` is not in our filenames, so the DB cannot be built from
 names alone — but one listing pass (the pass we make anyway) maps every live
 item to its `media_id`, and the archive's *file listing* says which we already
 hold. No extra Instagram requests, and no archive content — a listing is
 enough, which `GET /api/archives/:name/files` already serves.
 Measured on `0ct0ber19`: 2275 live media items, 2248 seeded from the existing
 listing, **27 left to download** — precisely the media of the two posts added
 since the last crawl.
 The one trap, which silently seeds almost nothing if you get it backwards:
 | surface | filed under | why |
 |---|---|---|
 | posts, reels | `post_shortcode` | carousel children each have their own `shortcode`, which never appears in a filename |
 | stories, highlights | `shortcode` (per item) | `post_shortcode` is the containing reel's id, shared by every item |
 `live_key()` encodes this. Matching on the wrong field seeded 5 of 2275.
 ### The skip-archive saves the CDN, not `instagram.com`
 Worth being exact about, because the two costs land on different surfaces and
 only one of them bans accounts:
 | what | which surface | scales with |
 |---|---|---|
 | downloading media | `scontent-*.cdninstagram.com` | how much is **new** |
 | enumerating the profile to find it | `instagram.com` | how **big** the profile is |
 The skip-archive suppresses the first. It does nothing about the second, so a
 2275-post profile costs ~76 pages of pagination every run, forever, whether it
 has three new posts or none. Seeding (above) saved a *second* full pass, not
 the first.
 Measured on the 2026-08-20 run, from sidecar write times in staging — free,
 since the run was paying for the listing anyway:
 ```
 1787248852   2026-08-19 … DcOeoVxkthi     new, +0s
 1787248944   2026-08-18 … DcLpfoJCZtp     new, +92s
 1787249058   2026-08-17 … DcIlGbxCUk0     new, +114s
 1787249162   2026-07-24 … DbKr1TxlPSX     ┐ all one second: nothing
 1787249162   2026-08-15 … DcD-FdBCYGm     ┘ downloaded, sidecars only
 ```
 Three posts took ~100s each; the remaining 2272 were enumeration with nothing
 to show for it.
 **Pinned posts do not break early abort.** Test case 16 previously claimed
 `0ct0ber19` returns its 3 pinned posts out of date order — that is true of the
 *web grid*, but the REST `/posts/` listing came back strictly
 reverse-chronological, newest first, no hoisting. That matters because
 front-loaded old posts are the one thing that would make `skip: abort:N`
 dangerous: it would trip on them and abort before reaching anything new.
 So `skip: abort:N` is viable, and cuts ~420 requests per run to ~40-60:
 | surface | live items | pages | with `abort:50` |
 |---|---:|---:|---:|
 | posts, 6 profiles | 11,248 | ~377 | ~12 |
 | reels, 6 profiles | 1,080 | ~24 | ~8 |
 | stories + highlights | — | ~20 | ~20 |
 N counts consecutive skipped **files**, not posts, so it must clear the largest
 already-held carousel — `DcD-FdBCYGm` alone is 22 media. 50 is comfortable; 5
 would not be.
 **The tradeoff is edited carousels.** Test case 15 is a post that gained items
 after we archived it, and only a full enumeration finds those. Suggested
 policy: `abort:50` for routine runs, a full sweep occasionally.
 Measured the same day, resuming a stopped run with `--abort 50`:
 | source | live items | enumerated |
 |---|---:|---:|
 | `cher_ryppo` posts | 2,151 | **7** |
 | `cher_ryppo` reels | 92 | 53 |
 One page instead of 72, and every new post was still caught. The 7 is roughly
 3 new posts plus 4 already-held carousels making up the 50 skipped files.
 Reels need 53 because they are single-media, so 50 consecutive skips really is
 50 reels — another reminder that N counts files, and that the same N behaves
 very differently on a carousel-heavy surface than on a reels tab.
 ## Publishing
 The fetch host stages to local disk and rsyncs afterwards. `rsync
 --ignore-existing` is not an optimisation but the safety property: the archive
 deliberately outlives Instagram, so publishing must only ever **add**. No
 `--delete`, and nothing already present is overwritten — including sidecars,
 which are rewritten every run and would otherwise churn the synced share.
 Publishing happens once at the end of a run, so a profile that fails midway
 never reaches the archive half-written.
 ## Status
 In use for all six ARTMS profiles.
 `withaseul` first — 322 files added (74 media, 241 `.json`, 7 `.txt`), nothing
 overwritten or deleted. Of the 74 new media, **zero** duplicated media already
 held under a different name, which is the check that says JD2 and gallery-dl
 naming really do converge.
 **2026-08-20**, the first full incremental sync, four days after the previous
 one. 184 new media, 299 files published, 0 failures and **0 CDN 429s**:
 | profile | posts | reels | stories | files added |
 |---|---:|---:|---:|---:|
 | 0ct0ber19 | 58 | 2 | 4 | +77 |
 | official_artms | 12 | — | 2 | +85 |
 | cher_ryppo | 41 | 1 | 8 | +63 |
 | zindoriyam | 23 | — | 4 | +35 |
 | kimxxlip | 16 | — | 2 | +23 |
 | withaseul | 10 | — | — | +16 |
 The 20 story items are the part that could not have been recovered later.
 Two things made it cheap, and both are worth keeping:
 - The archive DB was already seeded from the previous run, so `--min-interval`
  and the recorded `seeded` state meant **no probe passes at all**. A state
  file has to exist for this; if one is missing after a manual run, write it
  rather than letting the tool re-seed 24 sources.
 - `--abort 50` (see above) cut the remaining listing cost by roughly 85%.
 The run was deliberately **stopped and resumed** halfway to pick up `--abort`.
 That is safe precisely because of the state file: the 12 finished sources were
 already marked `fetched`, so the 20h floor skipped them and only the remaining
 12 re-ran. Stopping a run is cheap now; it was not before.
 Published files land owned by the SSH user rather than `rslsync`. The viewer
 reads them fine (world-readable), but Resilio does not own what it syncs; worth
 a `chown` if that ever matters. This also makes **`rsync` exit 23**
 ("some files/attrs were not transferred") the *normal* outcome of a publish —
 it is the failed `chown`, not lost data. Confirm by re-running the same rsync
 with `--dry-run`: an empty file list means everything arrived.
 The profiles to fetch live in `artms_account_links.txt` at the archive root,
 passed with `--urls-file`.
 ## Verified run
 `withaseul`, all four surfaces, staged locally and published to a scratch
 directory before the live publish above:
 ```
 ==> withaseul / posts      seeded 915 of 984 live items
 ==> withaseul / reels      seeded  28 of  34 live items
 ==> withaseul / stories    no results   (none active)
 ==> withaseul / highlights no results
 ```
 Output landed correctly, including the collab-reel case — `withaseul - reels`
 contains 53 files owned by `withaseul`, 10 by `cher_ryppo`, 3 by `0ct0ber19`
 and 2 by `official_artms`, all with the owner in the filename and the crawl
 scope as the directory.
 ### The CDN rate-limits, and the first run tripped it
 At `rate: 3M` with `sleep: [1.0, 3.0]`, `scontent-*.cdninstagram.com` returned
 **`429 Too Many Requests`** and two videos were lost (gallery-dl retried, then
 gave up with exit 4). This is the *tolerant* surface complaining, which is a
 clear signal the pacing was too aggressive.
 Defaults are now:
 | option | value |
 |---|---|
 | `--rate` | `1M` |
 | `--sleep-request` | 6–10 s |
 | `--sleep` | 3–6 s |
 | `sleep-429` | 120 s |
 | `retries` (extractor and downloader) | 8 |
 Re-running with those recovered both videos and produced **0 failures and 0
 429s**. Do not raise them for speed; an archive sync has no deadline.
 ### yt-dlp is worth installing
 Without it, gallery-dl logs `Cannot import yt-dlp or youtube-dl` and falls back
 to a progressive URL for DASH videos. The fallback mostly works but is what the
 429s hit hardest.
 **`pipx install yt-dlp` does not work** — it was the advice here until
 2026-08-20, and it is wrong. It gives yt-dlp its own venv, so the binary lands
 on `PATH` while gallery-dl, in a *different* venv, still cannot `import yt_dlp`.
 The symptom is that everything looks installed and the log keeps saying
 `Cannot import yt-dlp`. gallery-dl needs it importable, not runnable:
 ```sh
 pipx inject gallery-dl yt-dlp
 ```
 Verify by asking gallery-dl's own interpreter, not the shell:
 ```sh
 /home/matt/.local/share/pipx/venvs/gallery-dl/bin/python -c 'import yt_dlp'
 ```
 ## Known quirks
 - **`count` is not the emitted file count.** For 135 of 214 posts it was exactly
  one higher than the number of files written. This makes the `count >= 10`
  padding condition mis-pad a handful of 9-item posts (10 of 214 measured). Since
  the parser normalises the index, this is cosmetic — but it means a re-fetch
  over an existing JD2 tree writes `- 01.jpg` beside an existing `- 1.jpg`.
 - **Carousels get edited.** Two posts had a different media count live than on
  disk. Padding width follows the count *at download time*, so a grown carousel
  produces mixed widths — the archive already contains one such post from JD2.
 - **Highlights already have two naming styles on disk**, and every undated file
  has a dated twin. The scanner dedupes by index so they render once; it is
  wasted disk, not a display bug.
 ## Scanner work — done
 Shipped in `53b1f80` ("read gallery-dl sidecars for reel type and post dates").
 `useArchiveScanner` tells the three `.json` shapes apart **structurally**, not
 by filename, in `src/lib/gallery-dl-sidecar.ts`:
 1. Instagram export manifests (`posts_1.json`) — top-level `media` array.
 2. Instaloader `.json.xz` — GraphQL node under `node` / `__typename`.
 3. gallery-dl `.json` — flat, `post_shortcode` + `type`, none of the above.
 `post.isReel` now comes from the sidecar's `type`, which is Instagram's own
 classification, and beats every fallback in `post-tabs.ts`. Post dates are
 ranked rather than last-write-wins (`src/lib/post-dates.ts`): sidecar beats
 filename beats mtime.
 Note those files live on **`main`** — they parse the archive at display time
 and are viewer code, not fetching tooling.
 ## Test cases
 Real subjects, all present in the archive today. See
 `scripts/gdl-sync.py --selftest` for the harness.
 | # | case | shortcode | expected |
 |---|---|---|---|
 | 1 | single image | `CwcXnQhOqFG` | one `.jpg`, no index |
 | 2 | single feed video | `DbdG9L9jU4m` | one `.mp4`, `type: post` |
 | 3 | carousel, images only | `Cq8LrxSJAJE` | `- 1 … - 3` |
 | 4 | carousel, image + video | `CtohvHxLnWO` | `- 1.jpg … - 4.mp4`, **no `.txt`** |
 | 5 | carousel of exactly 9 | `Cv2Hb_brx_N` | 1-digit index |
 | 6 | carousel of 10+ | `CzM8Uf6B6H_` | 2-digit index `- 01 … - 10` |
 | 7 | reel shown on the posts grid | `C8FHM6EJl15` | in `<user>`, `type: reel` |
 | 8 | reel on the reels tab | `Db-lNCoib9m` | in `<user> - reels`, `type: reel` |
 | 9 | collab reel (other owner) | `DYcZOb0h6Sv` | dir `0ct0ber19 - reels`, filename `chuuo3o` |
 | 10 | story | live only | `story - <user>`, per-item shortcode + date |
 | 11 | story highlight | `C-IImhvpFuk` | `story highlights - <user> - <title>` |
 | 12 | highlight, unicode title | `Drawheeing⠀` | trailing U+2800 preserved in dirname |
 | 13 | empty caption | `CrdsY5CrSsO` | media written, `.txt` absent |
 | 14 | deleted post | `C0TgI7sphfZ` | on disk, absent live — must not be removed |
 | 15 | edited carousel | `C7zG7-jJMlq` | 18 on disk, 8 live — must not be removed |
 | 16 | pinned posts | `0ct0ber19` | REST listing is strictly reverse-chronological; see below |
 | 17 | profile avatar | `0ct0ber19.jpg` | base dir, undated |
 Cases 14–16 are reconciliation, not naming: **a sync must never delete**, since
 the archive deliberately outlives Instagram.
 Not covered, decide before relying on them: the `/reposts/` tab (`0ct0ber19`
 has one) and `/tagged/`. Neither is fetched today.
@@ -0,0 +1,194 @@
 # JDownloader2 — archive fetching
 > **This file lives only on the `tooling` branch.** `main` is published to
 > GitHub and deliberately carries none of this — not the host details, not the
 > IPs, and not the account names. `main`'s history was redacted on 2026-08-20;
 > real names exist only here.
 >
 > There is no `npm run jd2` script — `package.json` and `CLAUDE.md` are kept
 > byte-identical to `main` so that merging `main` into `tooling` never
 > conflicts. Run the crawljob generator directly:
 >
 > ```sh
 > npx tsx scripts/jd2-sync.ts --archives <dir> --dry-run
 > ```
 How content gets into this archive, and why the setup is shaped the way it is.
 ## Why JDownloader and not Instaloader
 There are two surfaces, and they're treated very differently:
 | Surface | What hits it | Risk |
 |---|---|---|
 | `instagram.com` | profile pages, GraphQL/API metadata | Tied to your session, heavily rate-limited. **This is where bans come from.** |
 | `scontent*.cdninstagram.com` | the actual media | Signed URLs, CDN-served, tolerant. Mostly a bandwidth question. |
 JDownloader does nearly all its work on the CDN. Instaloader's value — the rich
 `.json.xz` metadata — comes from asking `instagram.com` a question *per post*.
 Concretely, from this archive: `rivvsofficial` has 188 post-metadata files, so
 backfilling it cost 188 API requests for one 605-file profile. That's the ban
 vector. Downloading the 238 photos was never the problem.
 Instaloader got this account banned once. JDownloader with throttling did not.
 > **The account was suspended anyway, on 2026-08-17, for "spam".** Not by
 > JDownloader, and not by downloading. It was suspended during a day of
 > *building and verifying* the gallery-dl replacement — automated browser
 > scrolling to enumerate profile grids, repeated `--simulate` and `-j` metadata
 > passes, and one aborted sync that re-ran every listing pass before dying.
 >
 > The framing above is right about which surface is dangerous and wrong about
 > what reaches it. **Every read of `instagram.com` counts, including the ones
 > that download nothing** — and read-only work is easy not to count precisely
 > because it leaves no files behind. See the post-mortem at the top of
 > `docs/gallery-dl.md`.
 >
 > The rule that would have prevented it: *verify against the archive, never
 > against the live site*, and treat the first CDN `429` as the end of the
 > session rather than a pacing knob.
 ### What the metadata gap actually costs
 Comparing a JDownloader profile against an Instaloader one:
 | | JDownloader | Instaloader |
 |---|---|---|
 | Media | ✅ | ✅ |
 | Captions (`.txt`) | ✅ | ✅ |
 | Dates (from filenames) | ✅ | ✅ |
 | Bio / full name | ❌ | ✅ |
 | Follower counts | ❌ | ✅ |
 | External URL | ❌ | ✅ |
 Captions already work — the viewer reads the `.txt` sidecars. Everything missing
 lives in a *single* profile-level record, not the per-post ones. That's why
 JDownloader-sourced profiles show "0 followers" and a placeholder bio.
 Not worth extra requests. If you ever want it, the zero-request option is a
 hand-written `profile.json` sidecar (not implemented yet — ask).
 ## Settings that matter
 **Chunks per download → 1.** The single most important one. JDownloader splits
 each file into multiple ranged requests by default; that `Range` pattern looks
 nothing like a browser or the app. One chunk = one sequential GET per file.
 `jd2-sync` sets `chunks=1` per job, so no global change is needed — but set it
 globally too if you ever add links by hand.
 **Max simultaneous downloads → 2–3**, connections-per-host low. Concurrency is
 what turns "a user" into a statistic.
 **Leave reconnect / IP-change features off.** A mid-session IP change on a live
 cookie is a *stronger* anomaly signal than the request rate you'd be avoiding.
 ## The cookie
 Exported manually from a real browser session. This is the right approach — no
 programmatic login anywhere, which is the thing that actually gets flagged.
 - Use it from the **same public IP** as the browser it came from. A cookie used
  from a different network is what session-hijack detection looks for.
 - When it expires, **re-export from the browser**. Never add a login step to a tool.
 - It's a full account credential. Keep it off the NAS share and out of the repo.
 ## Workflow
 Two URLs per profile, because the profile grid misses some reels:
 ```
 https://www.instagram.com/<user>/
 https://www.instagram.com/<user>/reels/
 ```
 They overlap slightly — a reel caught by both lands in each directory and shows
 up twice in the viewer. That's correct and matches Instagram, which also shows
 reels in the profile grid *and* the Reels tab.
 ## Generating jobs
 Instead of pasting URLs and setting output folders by hand:
 ```bash
 npm run jd2 -- --archives /volume1/rslsync/sync/Instagram-archive/archives --dry-run
 ```
 Review, then write it into JDownloader's folder-watch directory:
 ```bash
 npm run jd2 -- --archives /volume1/rslsync/sync/Instagram-archive/archives \
               --out ~/.jd2/folderwatch
 ```
 JDownloader runs on the desktop while the archive lives on the NAS, so tell it
 the path *it* sees:
 ```bash
 npm run jd2 -- --archives /mnt/nas/Instagram-archive/archives \
               --download-base 'Z:\Instagram-archive\archives' \
               --out ~/.jd2/folderwatch
 ```
 | Flag | Purpose |
 |---|---|
 | `--archives <dir>` | Archive root to scan (or `$ARCHIVES_DIR`) |
 | `--out <dir>` | JDownloader folder-watch directory |
 | `--download-base <dir>` | Root path as JDownloader sees it (Windows paths fine) |
 | `--user <name>` | Just this profile (repeatable) |
 | `--skip <name>` | Never emit jobs for this directory (repeatable) |
 | `--chunks <n>` | Connections per file (default 1) |
 | `--auto-start` | Start immediately instead of parking in LinkGrabber |
 | `--all-reels` | Emit a reels job even where no reels directory exists |
 | `--dry-run` | Print instead of writing |
 Defaults are deliberately conservative: `chunks=1`, and links park in the
 LinkGrabber for review rather than auto-starting.
 Only posts and reels are emitted. Highlight URLs need a numeric id and story
 URLs expire, so those stay manual.
 Directories that aren't Instagram profiles are skipped by username shape
 (letters, digits, dots, underscores, ≤30 chars) — pointing a crawl at those
 spends `instagram.com` requests to be told the profile doesn't exist. For names
 that *look* like usernames but aren't, use `--skip` or a `.jd2ignore` file in
 the archive root, one name per line.
 Format reference: `src/org/jdownloader/extensions/folderwatchV2/explain.txt`.
 JDownloader develops on SVN — read it via the daily mirror at
 <https://github.com/mycodedoesnotcompile2/jdownloader_mirror> (`svn_trunk/`),
 not one of the abandoned GitHub copies.
 ## Expected layout
 Everything downloads into `<archives>/`, one directory per source:
 ```
 archives/
  0ct0ber19/                                 posts
  0ct0ber19 - reels/                         reels
  story - 0ct0ber19/                         stories
  story highlights - 0ct0ber19 - Heestory/   a highlight
 ```
 Non-archive directories (tool output, exports from elsewhere) live *outside*
 `archives/` so they never reach the viewer.
 The server picks up changes automatically — its index is keyed on directory
 mtime, so a new file invalidates only that directory.
 ## If something goes wrong
 **429 / rate limited** — stop for hours, not seconds. Retrying into a limit is
 what converts a soft throttle into something worse.
 **Cookie stops working** — re-export from the browser. Don't add a login step.
 **Files land in the wrong folder** — a Packagizer rule is overriding the job.
 Generated jobs set `overwritePackagizerEnabled=TRUE` to prevent this; check that
 rules aren't set to run after it.
 **Viewer doesn't show new posts** — check the file is in the right directory and
 matches the naming pattern (`YYYY-MM-DD_<user> - <shortcode>[ - NN].<ext>`).
 The index refreshes on directory mtime, so a genuinely new file is picked up on
 the next request.
@@ -1,12 +1,12 @@
 {
  "name": "instaarchive-viewer",
-  "version": "1.8.0",
+  "version": "1.8.1",
  "lockfileVersion": 3,
  "requires": true,
  "packages": {
    "": {
      "name": "instaarchive-viewer",
-      "version": "1.8.0",
+      "version": "1.8.1",
      "dependencies": {
        "@tailwindcss/vite": "^4.1.14",
        "@vitejs/plugin-react": "^5.0.4",
@@ -1,19 +1,18 @@
 {
  "name": "instaarchive-viewer",
  "private": true,
-  "version": "1.8.0",
+  "version": "1.8.1",
  "type": "module",
  "scripts": {
    "dev": "vite --port=3000 --host=0.0.0.0",
    "build": "vite build && npm run build:server",
-    "build:server": "tsc server.ts --esModuleInterop --module ESNext --target ES2022 --moduleResolution bundler --outDir dist-server",
+    "build:server": "tsc server.ts --esModuleInterop --module ESNext --target ES2022 --moduleResolution bundler --removeComments --outDir dist-server",
    "preview": "vite preview",
    "server": "tsx server.ts",
    "clean": "rm -rf dist",
    "lint": "tsc --noEmit",
    "test": "vitest run",
-    "test:watch": "vitest",
+    "test:watch": "vitest"
    "jd2": "tsx scripts/jd2-sync.ts"
  },
  "dependencies": {
    "@tailwindcss/vite": "^4.1.14",
@@ -0,0 +1,112 @@
 #!/bin/bash
 # Unattended wrapper around gdl-sync.py. One argument: the run mode.
 #
 #   stories     daily      ~6 requests; the only surface that cannot be backfilled
 #   profiles    monthly    every surface, --abort 50: stops enumerating a profile
 #                          once it reaches content already held, so it costs
 #                          ~40-60 requests and catches everything NEW
 #   full-sweep  rarely     every surface, no abort: walks each profile to the end
 #                          for ~420 requests. The only run that notices posts
 #                          EDITED after we archived them, and by far the most
 #                          expensive thing here -- see TOOLING.md before running.
 #
 # Exits non-zero if the sync does, so cron mails you. Everything is logged.
 set -eu
 MODE="${1:?usage: gdl-cron.sh stories|profiles|full-sweep}"
 GDL_HOME="${GDL_HOME:-$HOME/gdl}"
 INDEX="${GDL_INDEX:-https://instaarchive.ergosteur.com}"
 PUBLISH="${GDL_PUBLISH:-agentapi@10.20.28.200:/volume1/rslsync/sync/Instagram-archive/archives/}"
 STAGING="$GDL_HOME/staging-$MODE"
 # $$ in the name because two runs in the same SECOND would otherwise share a
 # log file, and `tee -a` appends -- which made a test see the previous run.
 LOG="$GDL_HOME/logs/$MODE-$(date +%Y%m%d-%H%M%S)-$$.log"
 PATH="$HOME/.local/bin:$PATH"; export PATH
 # Pacing. These are the values the 2026-08-22 runs used by hand, after the
 # scraping warning -- roughly double the caution of gdl-sync.py's own defaults
 # (6-10s / 3-6s / 1M). An archive sync has no deadline; being slow is free and
 # being restricted is not. Override per-run with GDL_SLEEP_REQUEST etc. if you
 # ever need to, but raise them rather than lower them.
 SLEEP_REQUEST="${GDL_SLEEP_REQUEST:-12 20}"
 SLEEP="${GDL_SLEEP:-5 10}"
 RATE="${GDL_RATE:-500K}"
 case "$MODE" in
  # --min-interval 8, not the 20h default. The floor exists to stop an ABORTED
  # RESTART re-enumerating profiles -- a minutes-to-hours concern. At 20h the
  # daily timer silently did nothing whenever a manual run had happened the
  # previous afternoon, which is exactly what happened on 2026-08-21: it fired,
  # skipped all six sources and reported success. A stories fetch is one
  # request per profile, so the worst case 8h permits is roughly twelve
  # requests in a day instead of six.
  stories)    ARGS="--only stories --min-interval 8" ;;
  profiles)   ARGS="--only posts,reels,stories,highlights --abort 50" ;;
  # No --abort: the whole point of full-sweep is enumerating to the end, so it
  # is the only run that notices carousels edited after we archived them.
  full-sweep) ARGS="--only posts,reels,stories,highlights" ;;
  # Renamed 2026-08-22: "full" was misleading (it is the abort-LIMITED run) and
  # "sweep" did not say it was the exhaustive one. Catch the old names rather
  # than failing with a bare error, in case something still passes them.
  full)  echo "mode 'full' was renamed to 'profiles'" >&2; exit 2 ;;
  sweep) echo "mode 'sweep' was renamed to 'full-sweep'" >&2; exit 2 ;;
  *) echo "unknown mode: $MODE (want stories|profiles|full-sweep)" >&2; exit 2 ;;
 esac
 mkdir -p "$GDL_HOME/logs"
 # Staging is wiped every run ON PURPOSE. What we already hold is decided by the
 # skip-archive (--download-archive), never by which files happen to be sitting
 # in staging, so starting empty is correct -- and it keeps the publish rsync
 # to just the new files instead of re-walking gigabytes each time.
 rm -rf "$STAGING"
 echo "=== $MODE run $(date -Is) ===" | tee -a "$LOG"
 # The exit status has to survive the pipe into tee. The left-hand side of a
 # pipeline runs in a SUBSHELL, so an `exit` in there sets the subshell's status
 # and the script goes on to return tee's, which is always 0. An earlier version
 # of this file did exactly that and reported success no matter what the sync
 # did -- which is why the shebang is bash: PIPESTATUS is the fix.
 # This run's output only. The check below must never see a previous run's
 # lines, so it reads this rather than the (appended-to) log.
 RUNOUT=$(mktemp)
 trap 'rm -f "$RUNOUT"' EXIT
 set +e
 # shellcheck disable=SC2086
 "$GDL_HOME/gdl-sync.py" \
    --index "$INDEX" \
    --staging "$STAGING" \
    --publish "$PUBLISH" \
    --archive-db "$GDL_HOME/artms.db" \
    --urls-file "$GDL_HOME/artms_account_links.txt" \
    --sleep-request $SLEEP_REQUEST \
    --sleep $SLEEP \
    --rate "$RATE" \
    $ARGS --execute 2>&1 | tee -a "$LOG" "$RUNOUT"
 status=${PIPESTATUS[0]}
 set -e
 # A stories run that skipped every source is NOT a success. It means the
 # min-interval floor blocked the one surface that cannot be backfilled, and
 # without this it looks identical to a clean run: exit 0, "0 step(s) failed".
 # Note this is not the same as "no stories today" -- that shows up as sources
 # being fetched and returning no results, which is normal and stays quiet.
 if [ "$MODE" = "stories" ] && grep -q "sources  : 0 to sync" "$RUNOUT"; then
  echo "WARNING: every stories source was skipped by --min-interval." | tee -a "$LOG" >&2
  echo "         Nothing was fetched. Stories expire in 24h and cannot be" | tee -a "$LOG" >&2
  echo "         backfilled, so this is a real loss, not a quiet no-op." | tee -a "$LOG" >&2
  [ "$status" -eq 0 ] && status=75
 fi
 echo "=== exit $status at $(date -Is) ===" | tee -a "$LOG"
 # Keep the log directory from growing without bound.
 ls -1t "$GDL_HOME/logs" | tail -n +30 | while read -r old; do
  rm -f "$GDL_HOME/logs/$old"
 done
 exit $status
@@ -0,0 +1,277 @@
 /**
 * Generate JDownloader2 .crawljob files for the archives on disk.
 *
 * The manual flow is: paste a profile URL into JDownloader, paste the /reels
 * URL separately (the profile page misses some reels), and set the output
 * folder by hand — times however many profiles you keep. This emits one
 * crawljob per source with the folder already pointed at the right directory,
 * so JDownloader's folder-watch picks the whole batch up at once.
 *
 * Profiles and their sidecar directories are derived with the same grouping
 * logic the server uses, so the output folders always match what the viewer
 * expects to find.
 *
 * Only posts and reels are emitted. Story and highlight URLs can't be rebuilt
 * from a directory name — highlights need their numeric id and stories expire —
 * so those stay manual.
 *
 * Crawljob format verified against JDownloader's own docs for the extension:
 * src/org/jdownloader/extensions/folderwatchV2/explain.txt. JDownloader
 * develops on SVN; read it via the daily mirror at
 * https://github.com/mycodedoesnotcompile2/jdownloader_mirror (svn_trunk/),
 * not one of the abandoned GitHub copies — several are a decade stale.
 *
 * Entries are separated by `->NEW ENTRY<-` and any property may be omitted.
 * There is also a `setBeforePackagizerEnabled` companion to
 * `overwritePackagizerEnabled`, if the Packagizer ever needs to see these
 * values before they're applied.
 *
 * Usage:
 *   npx tsx scripts/jd2-sync.ts --archives <dir> [options]
 *
 *   --archives <dir>       Archive root to scan (default: $ARCHIVES_DIR)
 *   --out <dir>            JDownloader folder-watch directory to write into
 *   --download-base <dir>  Root path as *JDownloader* sees it, when it runs on
 *                          a different machine than this script (e.g. a mapped
 *                          drive). Defaults to --archives.
 *   --user <name>          Only this profile (repeatable)
 *   --skip <name>          Never emit jobs for this directory (repeatable).
 *                          Also read from a `.jd2ignore` file in the archive
 *                          root, one name per line.
 *   --chunks <n>           Connections per file (default 1: multi-chunk ranged
 *                          requests are the one CDN pattern that doesn't look
 *                          like a browser)
 *   --auto-start           Start downloads immediately instead of parking them
 *                          in the LinkGrabber for review
 *   --all-reels            Emit a reels job even where no reels directory
 *                          exists yet
 *   --dry-run              Print the crawljob instead of writing it
 */
 import fs from 'fs';
 import path from 'path';
 import { groupArchiveDirectories, ArchiveSource } from '../src/lib/archive-grouping.js';
 interface Options {
  archives: string;
  out: string | null;
  downloadBase: string;
  users: string[];
  skip: Set<string>;
  chunks: number;
  autoStart: boolean;
  allReels: boolean;
  dryRun: boolean;
 }
 const parseArgs = (argv: string[]): Options => {
  const opts: Options = {
    archives: process.env.ARCHIVES_DIR ?? '',
    out: null,
    downloadBase: '',
    users: [],
    skip: new Set(),
    chunks: 1,
    autoStart: false,
    allReels: false,
    dryRun: false,
  };
  for (let i = 0; i < argv.length; i++) {
    const arg = argv[i];
    const next = () => argv[++i];
    switch (arg) {
      case '--archives': opts.archives = path.resolve(next()); break;
      case '--out': opts.out = path.resolve(next()); break;
      case '--download-base': opts.downloadBase = next(); break;
      case '--user': opts.users.push(next()); break;
      case '--skip': opts.skip.add(next()); break;
      case '--chunks': opts.chunks = parseInt(next(), 10); break;
      case '--auto-start': opts.autoStart = true; break;
      case '--all-reels': opts.allReels = true; break;
      case '--dry-run': opts.dryRun = true; break;
      case '--help': case '-h': printUsage(); process.exit(0);
      default:
        console.error(`Unknown argument: ${arg}`);
        process.exit(1);
    }
  }
  if (!opts.archives) {
    console.error('No archive root. Pass --archives <dir> or set ARCHIVES_DIR.');
    process.exit(1);
  }
  if (!opts.downloadBase) opts.downloadBase = opts.archives;
  if (!opts.out && !opts.dryRun) {
    console.error('No destination. Pass --out <folder-watch dir>, or --dry-run to preview.');
    process.exit(1);
  }
  return opts;
 };
 const printUsage = () => {
  const header = readHeaderComment();
  console.log(header);
 };
 /** Print the usage block from this file's own header comment. */
 const readHeaderComment = () => {
  try {
    const self = fs.readFileSync(new URL(import.meta.url), 'utf8');
    const usage = self.slice(self.indexOf(' * Usage:'), self.indexOf(' */'));
    return usage.split('\n').map(l => l.replace(/^ \* ?/, '')).join('\n');
  } catch {
    return 'See the comment at the top of scripts/jd2-sync.ts';
  }
 };
 /**
 * JDownloader escapes nothing in crawljob values, so a stray newline would
 * silently split a property. Paths with spaces are fine as-is.
 */
 const sanitise = (value: string) => value.replace(/[\r\n]+/g, ' ').trim();
 /**
 * Instagram usernames are 1–30 characters of letters, digits, dots and
 * underscores. Archive roots also collect directories that aren't profiles at
 * all — tool output, exports from other services — and pointing a crawl at
 * those spends requests on instagram.com to be told the profile doesn't exist.
 * That's the exact traffic worth not spending.
 */
 const USERNAME_RE = /^[A-Za-z0-9._]{1,30}$/;
 /** Directory names to skip, from `.jd2ignore` in the archive root. */
 const readIgnoreFile = (archives: string): string[] => {
  try {
    return fs.readFileSync(path.join(archives, '.jd2ignore'), 'utf8')
      .split('\n').map(l => l.trim()).filter(l => l && !l.startsWith('#'));
  } catch {
    return [];
  }
 };
 interface Job {
  user: string;
  kind: 'posts' | 'reels';
  url: string;
  packageName: string;
  downloadFolder: string;
  fileCount: number | null;
 }
 const buildJobs = (opts: Options): Job[] => {
  const dirNames = fs.readdirSync(opts.archives, { withFileTypes: true })
    .filter(e => e.isDirectory() && !/^[.@_]/.test(e.name))
    .map(e => e.name);
  const groups = groupArchiveDirectories(dirNames);
  const jobs: Job[] = [];
  const skipped: string[] = [];
  for (const name of readIgnoreFile(opts.archives)) opts.skip.add(name);
  const countFiles = (dir: string): number | null => {
    try {
      return fs.readdirSync(path.join(opts.archives, dir)).length;
    } catch {
      return null;
    }
  };
  // JDownloader must be given the path *it* can see, which differs from the
  // scan path whenever the archive lives on a share.
  const downloadFolderFor = (dir: string) =>
    opts.downloadBase.includes('\\')
      ? `${opts.downloadBase.replace(/\\$/, '')}\\${dir}`
      : path.posix.join(opts.downloadBase, dir);
  for (const [user, sources] of [...groups].sort(([a], [b]) => a.localeCompare(b))) {
    if (opts.users.length && !opts.users.includes(user)) continue;
    if (opts.skip.has(user)) { skipped.push(`${user} (ignored)`); continue; }
    if (!USERNAME_RE.test(user)) { skipped.push(`${user} (not a username)`); continue; }
    const has = (kind: ArchiveSource['kind']) => sources.find(s => s.kind === kind);
    const base = has('posts');
    if (!base) continue; // sidecar-only group: nothing sensible to point a URL at
    jobs.push({
      user, kind: 'posts',
      url: `https://www.instagram.com/${encodeURIComponent(user)}/`,
      packageName: base.dir,
      downloadFolder: downloadFolderFor(base.dir),
      fileCount: countFiles(base.dir),
    });
    const reels = has('reels');
    if (reels || opts.allReels) {
      const dir = reels?.dir ?? `${user} - reels`;
      jobs.push({
        user, kind: 'reels',
        url: `https://www.instagram.com/${encodeURIComponent(user)}/reels/`,
        packageName: dir,
        downloadFolder: downloadFolderFor(dir),
        fileCount: reels ? countFiles(dir) : null,
      });
    }
  }
  if (skipped.length) {
    console.error(`Skipped ${skipped.length} director${skipped.length === 1 ? 'y' : 'ies'}:`);
    for (const s of skipped) console.error(`  - ${s}`);
    console.error('');
  }
  return jobs;
 };
 const renderCrawljob = (jobs: Job[], opts: Options): string =>
  jobs.map(job => [
    `text=${sanitise(job.url)}`,
    `packageName=${sanitise(job.packageName)}`,
    `downloadFolder=${sanitise(job.downloadFolder)}`,
    `chunks=${opts.chunks}`,
    // Without this a Packagizer rule can override downloadFolder and scatter
    // files away from the directory the viewer reads.
    'overwritePackagizerEnabled=TRUE',
    `autoStart=${opts.autoStart ? 'TRUE' : 'FALSE'}`,
    `autoConfirm=${opts.autoStart ? 'TRUE' : 'FALSE'}`,
    'enabled=TRUE',
    `comment=instaarchive jd2-sync (${job.kind})`,
  ].join('\n')).join('\n->NEW ENTRY<-\n');
 const main = () => {
  const opts = parseArgs(process.argv.slice(2));
  const jobs = buildJobs(opts);
  if (!jobs.length) {
    console.error('No profiles matched.');
    process.exit(1);
  }
  console.error(`Archive root : ${opts.archives}`);
  console.error(`JD sees root : ${opts.downloadBase}`);
  console.error(`Jobs         : ${jobs.length} (${new Set(jobs.map(j => j.user)).size} profiles)\n`);
  for (const job of jobs) {
    const count = job.fileCount === null ? 'new' : `${job.fileCount} files`;
    console.error(`  ${job.kind.padEnd(5)} ${job.user.padEnd(24)} -> ${job.packageName}  (${count})`);
  }
  console.error('');
  const body = renderCrawljob(jobs, opts);
  if (opts.dryRun || !opts.out) {
    console.log(body);
    return;
  }
  fs.mkdirSync(opts.out, { recursive: true });
  const file = path.join(opts.out, `instaarchive-${new Date().toISOString().replace(/[:.]/g, '-')}.crawljob`);
  fs.writeFileSync(file, body, 'utf8');
  console.error(`Wrote ${file}`);
  console.error(opts.autoStart
    ? 'Downloads will start automatically.'
    : 'Links land in the LinkGrabber for review; start them when ready.');
 };
 main();
@@ -0,0 +1,237 @@
 #!/usr/bin/env python3
 """
 Find a profile's reels by scrolling the real page, not calling the API.
 gallery-dl's dedicated reels extractor POSTs to /api/v1/clips/user/, which
 Instagram now 302-redirects to the home page for this account -- confirmed on
 2026-08-26/27 across multiple profiles, hours apart, with a freshly-warmed
 session and a correct X-IG-WWW-Claim header. The reels tab itself loads fine
 in a real browser, so this drives the SAME logged-in Chrome (already exposed
 on loopback CDP for MCP automation -- see TOOLING.md) via the DevTools
 protocol, scrolls it like a person would, and scrapes `/reel/<code>/` links
 out of the rendered page instead.
 This only finds shortcodes; it never downloads anything itself. What comes
 out the other end is deduped against the archive (via the same --index used
 elsewhere) and printed as plain post URLs -- feed them to gdl-sync.py:
    ./scripts/reels-scrape.py --profile someuser \\
        --index https://instaarchive.ergosteur.com > /tmp/someuser-reels.txt
    ./scripts/gdl-sync.py --publish user@host:/path --staging /var/tmp/gdl \\
        --post-urls-file /tmp/someuser-reels.txt \\
        --sleep-request 12 20 --sleep 5 10 --rate 500K --execute
 Requires the `websocket-client` package (only imported inside scrape_reels,
 so everything else here stays importable -- and testable -- without it).
 """
 from __future__ import annotations
 import argparse
 import itertools
 import json
 import random
 import re
 import sys
 import time
 from pathlib import Path
 from urllib.request import urlopen
 RE_REEL_HREF = re.compile(r"/reel/([^/?#]+)")
 # The reels tab has finished loading once a scroll round finds no NEW
 # shortcodes this many times in a row -- lazy-loaded feeds sometimes stall
 # for a round or two before producing more, so one dry round is not enough.
 DEFAULT_MAX_IDLE_ROUNDS = 3
 DEFAULT_MAX_SCROLLS = 200
 DEFAULT_SCROLL_PAUSE = (2.0, 3.5)
 SCRAPE_JS = (
    "(() => {"
    "window.scrollTo(0, document.body.scrollHeight);"
    "return Array.from(document.querySelectorAll('a[href*=\"/reel/\"]'))"
    ".map(a => a.getAttribute('href'));"
    "})()"
 )
 def extract_shortcodes(hrefs: list[str]) -> list[str]:
    codes = []
    for href in hrefs:
        if m := RE_REEL_HREF.search(href):
            codes.append(m.group(1))
    return codes
 class CDPError(RuntimeError):
    pass
 class CDP:
    """
    A deliberately minimal synchronous DevTools Protocol client: one request
    in flight at a time, which is all a linear scroll-and-scrape loop needs.
    Anything fancier (concurrent requests, event subscriptions) is scope this
    script has no reason to carry.
    """
    def __init__(self, ws_url: str, timeout: float = 30.0):
        import websocket  # local: keep this importable without the package
        self.ws = websocket.create_connection(ws_url, timeout=timeout)
        self._ids = itertools.count(1)
    def send(self, method: str, params: dict | None = None,
             session_id: str | None = None) -> dict:
        msg_id = next(self._ids)
        payload = {"id": msg_id, "method": method, "params": params or {}}
        if session_id:
            payload["sessionId"] = session_id
        self.ws.send(json.dumps(payload))
        while True:
            msg = json.loads(self.ws.recv())
            if msg.get("id") != msg_id:
                continue  # an event notification, not our reply -- ignore
            if "error" in msg:
                raise CDPError(f"{method}: {msg['error']}")
            return msg.get("result", {})
    def close(self) -> None:
        self.ws.close()
 def scrape_reels(user: str, cdp_port: int = 9222,
                 scroll_pause: tuple[float, float] = DEFAULT_SCROLL_PAUSE,
                 max_idle_rounds: int = DEFAULT_MAX_IDLE_ROUNDS,
                 max_scrolls: int = DEFAULT_MAX_SCROLLS,
                 log=lambda msg: None) -> list[str]:
    """
    Open the profile's reels tab in a NEW tab of the already-signed-in Chrome,
    scroll it to the bottom repeatedly, and collect every unique `/reel/`
    shortcode that appears. Closes the tab when done either way.
    """
    version = json.loads(urlopen(f"http://localhost:{cdp_port}/json/version",
                                 timeout=10).read())
    browser = CDP(version["webSocketDebuggerUrl"])
    target_id = None
    try:
        target = browser.send("Target.createTarget", {
            "url": f"https://www.instagram.com/{user}/reels/"})
        target_id = target["targetId"]
        attach = browser.send("Target.attachToTarget", {
            "targetId": target_id, "flatten": True})
        session_id = attach["sessionId"]
        browser.send("Page.enable", session_id=session_id)
        browser.send("Runtime.enable", session_id=session_id)
        time.sleep(4.0)  # initial page load, before the first scroll
        seen: set[str] = set()
        idle = 0
        for i in range(max_scrolls):
            result = browser.send(
                "Runtime.evaluate",
                {"expression": SCRAPE_JS, "returnByValue": True},
                session_id=session_id)
            hrefs = result.get("result", {}).get("value") or []
            codes = extract_shortcodes(hrefs)
            new = [c for c in codes if c not in seen]
            seen.update(new)
            log(f"  scroll {i + 1}: {len(seen)} unique reels so far (+{len(new)})")
            if new:
                idle = 0
            else:
                idle += 1
                if idle >= max_idle_rounds:
                    break
            time.sleep(random.uniform(*scroll_pause))
        return sorted(seen)
    finally:
        if target_id:
            try:
                browser.send("Target.closeTarget", {"targetId": target_id})
            except CDPError:
                pass  # best-effort cleanup; a leftover tab is harmless
        browser.close()
 def main() -> int:
    ap = argparse.ArgumentParser(description=__doc__,
                                 formatter_class=argparse.RawDescriptionHelpFormatter)
    ap.add_argument("--profile", required=True,
                    help="Instagram username to scrape reels for")
    ap.add_argument("--index", required=True,
                    help="existing archive listing, same as gdl-sync.py's "
                         "--index: a local root, or the viewer's base URL. "
                         "Used only to dedupe -- already-archived shortcodes "
                         "are dropped before anything is printed")
    ap.add_argument("--cdp-port", type=int, default=9222,
                    help="Chrome's loopback DevTools port (see TOOLING.md)")
    ap.add_argument("--scroll-pause", nargs=2, type=float,
                    default=list(DEFAULT_SCROLL_PAUSE), metavar=("MIN", "MAX"),
                    help="random pause between scrolls, seconds")
    ap.add_argument("--max-idle-rounds", type=int, default=DEFAULT_MAX_IDLE_ROUNDS,
                    help="stop after this many consecutive scrolls with no "
                         "new reels")
    ap.add_argument("--max-scrolls", type=int, default=DEFAULT_MAX_SCROLLS,
                    help="hard ceiling on scroll rounds, in case a page never "
                         "goes idle")
    ap.add_argument("--out", type=Path,
                    help="write new reel URLs here, one per line (default: "
                         "stdout)")
    args = ap.parse_args()
    import importlib.util
    spec = importlib.util.spec_from_file_location(
        "gdl_sync", Path(__file__).with_name("gdl-sync.py"))
    gdl = importlib.util.module_from_spec(spec)
    sys.modules["gdl_sync"] = gdl
    spec.loader.exec_module(gdl)
    index = gdl.ArchiveIndex(args.index)
    # Deduped against the WHOLE archive, not just this profile's own
    # directories: a shortcode is globally unique, and a reels tab commonly
    # surfaces reposts/collabs by OTHER tracked accounts. Missing that let a
    # 2026-08-27 run re-fetch 9 reels already held under their true owner's
    # directory -- gallery-dl filed them correctly there (by the post's real
    # `username`, not the scraped profile), so nothing was lost, but it spent
    # 9 avoidable Instagram requests to find that out. All of this is local
    # requests to our own viewer, never to instagram.com, so checking every
    # profile costs nothing on the budget that actually matters.
    profiles = index.profiles()
    have: set[str] = set()
    for profile in profiles:
        have.update(code for code, _ in gdl.index_existing(index.listing(profile)))
    print(f"archive already holds {len(have)} shortcode(s) across "
          f"{len(profiles)} profile(s)", file=sys.stderr)
    print(f"scraping https://www.instagram.com/{args.profile}/reels/ ...",
          file=sys.stderr)
    codes = scrape_reels(args.profile, cdp_port=args.cdp_port,
                        scroll_pause=tuple(args.scroll_pause),
                        max_idle_rounds=args.max_idle_rounds,
                        max_scrolls=args.max_scrolls,
                        log=lambda msg: print(msg, file=sys.stderr))
    new_codes = [c for c in codes if c not in have]
    print(f"found {len(codes)} reel(s) on the page, {len(codes) - len(new_codes)} "
          f"already archived, {len(new_codes)} new", file=sys.stderr)
    urls = [f"https://www.instagram.com/{args.profile}/reel/{c}/"
            for c in new_codes]
    text = "\n".join(urls)
    if args.out:
        args.out.write_text(text + ("\n" if text else ""))
        print(f"wrote {len(urls)} URL(s) to {args.out}", file=sys.stderr)
    else:
        if text:
            print(text)
    return 0
 if __name__ == "__main__":
    sys.exit(main())
@@ -0,0 +1,114 @@
 #!/bin/bash
 # The full reels pipeline for one profile: scrape by scrolling the real page,
 # then fetch and publish whatever's new. See TOOLING.md ("Reels: the API is
 # blocked, scrape by scrolling instead") for why this exists at all -- the
 # dedicated reels API is blocked, this drives the actual signed-in browser
 # session instead, and is slower by design.
 #
 # Usage: ./reels-sync.sh <username-or-profile-url>
 #   ./reels-sync.sh zindoriyam
 #   ./reels-sync.sh https://www.instagram.com/zindoriyam/
 #
 # Deliberately NOT wired into gdl-cron.sh or the timers -- see TOOLING.md.
 # Exits non-zero if either step does. Everything is logged.
 set -eu
 RAW="${1:?usage: reels-sync.sh <username-or-profile-url>}"
 GDL_HOME="${GDL_HOME:-$HOME/gdl}"
 GDL_PYTHON="${GDL_PYTHON:-$HOME/.local/share/pipx/venvs/gallery-dl/bin/python3}"
 INDEX="${GDL_INDEX:-https://instaarchive.ergosteur.com}"
 PUBLISH="${GDL_PUBLISH:-agentapi@10.20.28.200:/volume1/rslsync/sync/Instagram-archive/archives/}"
 # Same hand-paced pacing gdl-cron.sh uses -- see its comment for why. Override
 # per-run with GDL_SLEEP_REQUEST etc. if you ever need to, but raise them
 # rather than lower them.
 SLEEP_REQUEST="${GDL_SLEEP_REQUEST:-12 20}"
 SLEEP="${GDL_SLEEP:-5 10}"
 RATE="${GDL_RATE:-500K}"
 SCROLL_PAUSE="${GDL_SCROLL_PAUSE:-2.0 3.5}"
 MAX_IDLE_ROUNDS="${GDL_MAX_IDLE_ROUNDS:-3}"
 PATH="$HOME/.local/bin:$PATH"; export PATH
 if [ ! -x "$GDL_PYTHON" ]; then
  echo "reels-scrape.py needs gallery-dl's own pipx venv python (websocket-client" >&2
  echo "was injected there, not into the system python): $GDL_PYTHON not found" >&2
  exit 2
 fi
 # Same username-from-URL parsing gdl-sync.py already does for --urls-file,
 # reused rather than re-implemented so the two never drift apart.
 PROFILE=$("$GDL_PYTHON" -c "
 import re, sys, importlib.util
 from pathlib import Path
 spec = importlib.util.spec_from_file_location('gdl_sync', Path('$GDL_HOME/gdl-sync.py'))
 gdl = importlib.util.module_from_spec(spec)
 sys.modules['gdl_sync'] = gdl
 spec.loader.exec_module(gdl)
 raw = '$RAW'
 m = gdl.RE_PROFILE_URL.match(raw)
 user = m.group('user') if m else raw.strip('/')
 if not user or '/' in user or ' ' in user:
    print(f'cannot read a username from {raw!r}', file=sys.stderr)
    sys.exit(1)
 print(user)
 ")
 mkdir -p "$GDL_HOME/logs"
 LOG="$GDL_HOME/logs/reels-$PROFILE-$(date +%Y%m%d-%H%M%S)-$$.log"
 URLS_FILE="$GDL_HOME/$PROFILE-reels.txt"
 STAGING="$GDL_HOME/staging-reels-$PROFILE"
 echo "=== reels-sync $PROFILE $(date -Is) ===" | tee -a "$LOG"
 # The exit status has to survive the pipe into tee -- see gdl-cron.sh's
 # comment on PIPESTATUS for why this needs to be bash, not sh.
 set +e
 "$GDL_PYTHON" "$GDL_HOME/reels-scrape.py" \
    --profile "$PROFILE" \
    --index "$INDEX" \
    --scroll-pause $SCROLL_PAUSE \
    --max-idle-rounds "$MAX_IDLE_ROUNDS" \
    --out "$URLS_FILE" 2>&1 | tee -a "$LOG"
 scrape_status=${PIPESTATUS[0]}
 set -e
 if [ "$scrape_status" -ne 0 ]; then
  echo "=== exit $scrape_status (scrape failed) at $(date -Is) ===" | tee -a "$LOG"
  exit "$scrape_status"
 fi
 if [ ! -s "$URLS_FILE" ]; then
  echo "no new reels for $PROFILE; nothing to fetch" | tee -a "$LOG"
  echo "=== exit 0 at $(date -Is) ===" | tee -a "$LOG"
  exit 0
 fi
 # Staging is wiped every run on purpose -- same reasoning as gdl-cron.sh: what
 # we already hold is decided by the archive dedupe in reels-scrape.py, not by
 # what happens to be sitting in staging.
 rm -rf "$STAGING"
 set +e
 # shellcheck disable=SC2086
 "$GDL_HOME/gdl-sync.py" \
    --publish "$PUBLISH" \
    --staging "$STAGING" \
    --post-urls-file "$URLS_FILE" \
    --sleep-request $SLEEP_REQUEST \
    --sleep $SLEEP \
    --rate "$RATE" \
    --execute 2>&1 | tee -a "$LOG"
 status=${PIPESTATUS[0]}
 set -e
 echo "=== exit $status at $(date -Is) ===" | tee -a "$LOG"
 # Keep the log directory from growing without bound -- scoped to this
 # script's own logs so it never touches gdl-cron.sh's rotation.
 ls -1t "$GDL_HOME/logs" | grep '^reels-' | tail -n +30 | while read -r old; do
  rm -f "$GDL_HOME/logs/$old"
 done
 exit "$status"
@@ -0,0 +1,16 @@
 [Unit]
 # One templated service for all three modes; the instance name (%i) is the
 # mode: stories, full or sweep.
 Description=Instagram archive sync (%i)
 Documentation=file:%h/gdl/TOOLING.md
 After=network-online.target
 Wants=network-online.target
 [Service]
 Type=oneshot
 ExecStart=%h/gdl/gdl-cron.sh %i
 # A sync has no deadline and the pacing is deliberately slow; a full sweep can
 # run for hours. Never let systemd kill one midway -- a half-published run is
 # the one state the publish step is designed to avoid.
 TimeoutStartSec=infinity
 Nice=10
@@ -0,0 +1,18 @@
 [Unit]
 Description=Full Instagram archive sweep, no abort (~420 requests)
 # The only run that enumerates every profile to the end, and so the only one
 # that notices carousels edited after we archived them (test case 15).
 #
 # It is also by far the most expensive thing here: ~420 requests to
 # instagram.com, the same order as the run that preceded the 2026-08-21
 # scraping warning, spent to catch a handful of retroactively edited posts.
 # Consider running it by hand when you mean to, rather than on a timer.
 [Timer]
 # Month names are not valid in OnCalendar's date field -- numeric only.
 OnCalendar=*-01,04,07,10-07 04:00:00
 RandomizedDelaySec=45m
 Persistent=true
 [Install]
 WantedBy=timers.target
@@ -0,0 +1,10 @@
 [Unit]
 Description=Monthly Instagram profile sync (all surfaces, --abort 50)
 [Timer]
 OnCalendar=*-*-03 04:00:00
 RandomizedDelaySec=45m
 Persistent=true
 [Install]
 WantedBy=timers.target
@@ -0,0 +1,15 @@
 [Unit]
 Description=Daily Instagram stories sync
 # Stories expire in 24h and cannot be backfilled. This is the only timer whose
 # missed run means content is gone for good, which is what Persistent= is for.
 [Timer]
 OnCalendar=*-*-* 09:00:00
 # Not a fixed time: a job firing at exactly 09:00 every day is obviously a
 # machine, and the whole safety model is about not looking like one.
 RandomizedDelaySec=45m
 # Catch up after the host was asleep or off. cron would silently skip.
 Persistent=true
 [Install]
 WantedBy=timers.target
@@ -0,0 +1,272 @@
 #!/usr/bin/env python3
 """
 Tests for the request-budget logic in gdl-sync.py.
    python3 -m unittest discover -s scripts -p 'test_*.py'
 Deliberately stdlib-only, so it runs anywhere the sync itself runs. What is
 covered here is the part that decides whether to spend a request — the part
 whose absence got the archive's Instagram account suspended.
 """
 import datetime as dt
 import importlib.util
 import json
 import sys
 import tempfile
 import unittest
 from pathlib import Path
 _spec = importlib.util.spec_from_file_location(
    "gdl_sync", Path(__file__).with_name("gdl-sync.py"))
 gdl = importlib.util.module_from_spec(_spec)
 sys.modules["gdl_sync"] = gdl
 _spec.loader.exec_module(gdl)
 NOW = dt.datetime(2026, 8, 18, 12, 0, tzinfo=dt.timezone.utc)
 NOW_TS = NOW.timestamp()
 def ago(hours: float) -> str:
    return (NOW - dt.timedelta(hours=hours)).isoformat()
 class SourceSelection(unittest.TestCase):
    def test_only_stories_is_a_single_cheap_source(self):
        srcs = gdl.Profile("u").sources(gdl.STORIES_ONLY)
        self.assertEqual([s.kind for s in srcs], ["stories"])
        self.assertEqual(srcs[0].directory, "story - u")
    def test_full_sync_covers_every_surface(self):
        srcs = gdl.Profile("u").sources(set(gdl.ALL_KINDS))
        self.assertEqual([s.kind for s in srcs], list(gdl.ALL_KINDS))
    def test_reels_and_stories_go_to_their_own_directories(self):
        by_kind = {s.kind: s for s in gdl.Profile("u").sources(set(gdl.ALL_KINDS))}
        self.assertEqual(by_kind["posts"].directory, "u")
        self.assertEqual(by_kind["reels"].directory, "u - reels")
        # Highlights derive their directory from the title mid-extraction.
        self.assertEqual(by_kind["highlights"].directory, "")
 class PlanSource(unittest.TestCase):
    def setUp(self):
        self.tmp = tempfile.TemporaryDirectory()
        self.state = gdl.SyncState(Path(self.tmp.name) / "state.json")
        self.posts = gdl.Profile("u").sources({"posts"})[0]
        self.stories = gdl.Profile("u").sources({"stories"})[0]
    def tearDown(self):
        self.tmp.cleanup()
    def test_first_run_seeds(self):
        fetch, seed, _ = gdl.plan_source(self.posts, self.state, NOW_TS, 20)
        self.assertTrue(fetch)
        self.assertTrue(seed)
    def test_seeding_happens_only_once(self):
        self.state.mark_seeded(self.posts.url, ago(720))
        fetch, seed, reason = gdl.plan_source(self.posts, self.state, NOW_TS, 20)
        self.assertTrue(fetch)
        self.assertFalse(seed, "a seeded source must never be re-probed")
        self.assertIn("already seeded", reason)
    def test_stories_never_seed(self):
        # A story cannot be in the archive before it is fetched, so probing
        # would double the cost of the cheapest surface for no benefit.
        _, seed, reason = gdl.plan_source(self.stories, self.state, NOW_TS, 20)
        self.assertFalse(seed)
        self.assertIn("no seed", reason)
    def test_recent_fetch_is_refused(self):
        self.state.mark_fetched(self.posts.url, ago(3))
        fetch, _, reason = gdl.plan_source(self.posts, self.state, NOW_TS, 20)
        self.assertFalse(fetch)
        self.assertIn("under the", reason)
    def test_an_old_fetch_is_allowed_again(self):
        self.state.mark_fetched(self.posts.url, ago(30))
        fetch, _, _ = gdl.plan_source(self.posts, self.state, NOW_TS, 20)
        self.assertTrue(fetch)
    def test_daily_stories_pass_a_20h_floor(self):
        # The cadence this is built for: once a day, every day.
        self.state.mark_fetched(self.stories.url, ago(24))
        fetch, _, _ = gdl.plan_source(self.stories, self.state, NOW_TS, 20)
        self.assertTrue(fetch)
    def test_force_disables_the_floor(self):
        self.state.mark_fetched(self.posts.url, ago(1))
        fetch, _, _ = gdl.plan_source(self.posts, self.state, NOW_TS, 0.0)
        self.assertTrue(fetch)
    def test_the_aborted_run_scenario(self):
        """
        Yesterday's failure: a run died mid-way and the restart re-enumerated
        every profile. Seeded-but-not-fetched must not re-probe.
        """
        self.state.mark_seeded(self.posts.url, ago(0.5))
        fetch, seed, _ = gdl.plan_source(self.posts, self.state, NOW_TS, 20)
        self.assertTrue(fetch, "the fetch still needs to happen")
        self.assertFalse(seed, "but the listing pass must not be paid for twice")
 class StatePersistence(unittest.TestCase):
    def test_state_survives_a_reload(self):
        with tempfile.TemporaryDirectory() as d:
            path = Path(d) / "state.json"
            a = gdl.SyncState(path)
            a.mark_seeded("https://x/", ago(1))
            a.mark_fetched("https://x/", ago(1))
            a.save()
            b = gdl.SyncState(path)
            self.assertFalse(b.needs_seed("https://x/"))
            self.assertEqual(b.last_fetch("https://x/"), ago(1))
    def test_a_corrupt_state_file_never_blocks_a_sync(self):
        with tempfile.TemporaryDirectory() as d:
            path = Path(d) / "state.json"
            path.write_text("{ not json")
            self.assertTrue(gdl.SyncState(path).needs_seed("https://x/"))
 class ProbeCaching(unittest.TestCase):
    def test_fresh_entries_are_reused_and_stale_ones_are_not(self):
        with tempfile.TemporaryDirectory() as d:
            cache = gdl.ProbeCache(Path(d) / "p.json", ttl_hours=24)
            cache.put("https://x/", [{"shortcode": "A", "post_shortcode": "A",
                                      "num": 1, "media_id": "1"}], ago(1))
            self.assertEqual(len(cache.get("https://x/", NOW_TS)), 1)
            cache.put("https://y/", [{"shortcode": "B", "post_shortcode": "B",
                                      "num": 1, "media_id": "2"}], ago(48))
            self.assertIsNone(cache.get("https://y/", NOW_TS))
    def test_cache_keeps_only_the_fields_seeding_needs(self):
        with tempfile.TemporaryDirectory() as d:
            path = Path(d) / "p.json"
            cache = gdl.ProbeCache(path, ttl_hours=24)
            cache.put("https://x/", [{"shortcode": "A", "post_shortcode": "A",
                                      "num": 1, "media_id": "1",
                                      "description": "x" * 5000}], ago(0))
            cache.save()
            self.assertNotIn("description", path.read_text())
    def test_a_miss_is_reported_rather_than_guessed(self):
        with tempfile.TemporaryDirectory() as d:
            cache = gdl.ProbeCache(Path(d) / "p.json", ttl_hours=24)
            self.assertIsNone(cache.get("https://never-seen/", NOW_TS))
 class Seeding(unittest.TestCase):
    """The bug that seeded 5 of 2275: matching the wrong shortcode field."""
    def test_posts_are_keyed_by_post_shortcode(self):
        item = {"shortcode": "childcode", "post_shortcode": "POSTCODE",
                "num": 2, "media_id": "9"}
        self.assertEqual(gdl.live_key(item, "posts"), ("POSTCODE", 2))
    def test_stories_are_keyed_by_the_per_item_shortcode(self):
        item = {"shortcode": "ITEMCODE", "post_shortcode": "reelid",
                "num": 3, "media_id": "9"}
        self.assertEqual(gdl.live_key(item, "stories"), ("ITEMCODE", 1))
        self.assertEqual(gdl.live_key(item, "highlights"), ("ITEMCODE", 1))
    def test_index_existing_normalises_a_missing_index_to_one(self):
        held = gdl.index_existing([
            "u/2023-04-19_u - ABC.mp4",
            "u/2023-04-12_u - DEF - 3.jpg",
            "u/2023-04-12_u - DEF.txt",      # sidecars are not media
            "u/2023-04-12_u - DEF.json",
        ])
        self.assertEqual(held, {("ABC", 1), ("DEF", 3)})
    def test_seeding_marks_only_what_is_already_held(self):
        with tempfile.TemporaryDirectory() as d:
            db = Path(d) / "a.db"
            live = [
                {"post_shortcode": "HELD", "shortcode": "x", "num": 1, "media_id": "11"},
                {"post_shortcode": "NEW", "shortcode": "y", "num": 1, "media_id": "22"},
            ]
            n = gdl.seed_archive_db(db, {("HELD", 1)}, live, "posts")
            self.assertEqual(n, 1)
            import sqlite3
            rows = {r[0] for r in sqlite3.connect(db).execute(
                "SELECT entry FROM archive")}
            self.assertEqual(rows, {"instagram11"})
 class Publishing(unittest.TestCase):
    def test_publish_only_ever_adds(self):
        cmd = gdl.rsync_command(Path("/stage"), "host:/archives", dry_run=False)
        self.assertIn("--ignore-existing", cmd)
        self.assertNotIn("--delete", cmd)
    def test_tooling_files_are_excluded_from_the_archive(self):
        cmd = " ".join(gdl.rsync_command(Path("/stage"), "/dest", dry_run=True))
        for pattern in ("gdl-sync*.json", "*.db"):
            self.assertIn(pattern, cmd)
        self.assertIn("--dry-run", cmd)
 class PostUrl(unittest.TestCase):
    """--post-url: a one-off fetch outside the tracked profile list, whose
    owning account is only known mid-extraction -- same reasoning as
    highlights, so it must be exempted from the same forced-destination rule."""
    def test_config_keys_a_username_directory(self):
        config = gdl.build_config("1M", [6.0, 10.0], [3.0, 6.0])
        for kind in ("post", "reel"):
            self.assertEqual(
                config["extractor"]["instagram"][kind]["directory"],
                ["{username}"])
    def test_destination_is_not_forced_like_posts_and_reels(self):
        staging = Path("/stage")
        for subcategory in ("post", "reel"):
            src = gdl.Source(subcategory, "https://www.instagram.com/p/ABC/",
                             "", subcategory)
            cmd = gdl.gdl_command(src, staging, Path("/cfg.json"), "chrome:x",
                                  None)
            self.assertIn(str(staging), cmd)
            self.assertNotIn(str(staging / subcategory), cmd)
 class UrlsFile(unittest.TestCase):
    def test_reads_every_form_a_person_might_paste(self):
        with tempfile.TemporaryDirectory() as d:
            p = Path(d) / "urls.txt"
            p.write_text(
                "# comment\n"
                "https://www.instagram.com/a/\n"
                "https://instagram.com/b\n"
                "www.instagram.com/c/\n"
                "d\n"
                "  e   # trailing\n"
                "\n"
                "https://www.instagram.com/a/\n"        # duplicate
                "https://www.instagram.com/p/ABC123/\n" # a post, not a profile
                "not a username\n")
            self.assertEqual(gdl.read_urls_file(p), ["a", "b", "c", "d", "e"])
 class PostUrlsFile(unittest.TestCase):
    """The format reels-scrape.py writes: whole URLs, not usernames."""
    def test_skips_comments_blanks_and_duplicates(self):
        with tempfile.TemporaryDirectory() as d:
            p = Path(d) / "reels.txt"
            p.write_text(
                "# scraped 2026-08-27\n"
                "https://www.instagram.com/u/reel/AAA/\n"
                "\n"
                "https://www.instagram.com/u/reel/BBB/\n"
                "https://www.instagram.com/u/reel/AAA/\n")  # duplicate
            self.assertEqual(gdl.read_post_urls_file(p), [
                "https://www.instagram.com/u/reel/AAA/",
                "https://www.instagram.com/u/reel/BBB/",
            ])
 if __name__ == "__main__":
    unittest.main(verbosity=2)
@@ -0,0 +1,40 @@
 #!/usr/bin/env python3
 """
 Tests for the pure parts of reels-scrape.py -- everything except the actual
 CDP session, which needs a live signed-in Chrome and is exercised by hand.
    python3 -m unittest discover -s scripts -p 'test_*.py'
 """
 import importlib.util
 import sys
 import unittest
 from pathlib import Path
 _spec = importlib.util.spec_from_file_location(
    "reels_scrape", Path(__file__).with_name("reels-scrape.py"))
 scrape = importlib.util.module_from_spec(_spec)
 sys.modules["reels_scrape"] = scrape
 _spec.loader.exec_module(scrape)
 class ExtractShortcodes(unittest.TestCase):
    def test_reads_relative_and_absolute_hrefs(self):
        hrefs = [
            "/someuser/reel/ABC123/",
            "https://www.instagram.com/someuser/reel/DEF456/",
            "/reel/GHI789/?img_index=1",
        ]
        self.assertEqual(scrape.extract_shortcodes(hrefs),
                         ["ABC123", "DEF456", "GHI789"])
    def test_ignores_non_reel_links(self):
        hrefs = ["/someuser/", "/someuser/p/ABC123/", "/explore/tags/foo/"]
        self.assertEqual(scrape.extract_shortcodes(hrefs), [])
    def test_empty_input(self):
        self.assertEqual(scrape.extract_shortcodes([]), [])
 if __name__ == "__main__":
    unittest.main(verbosity=2)