# Manual run — scrape, archive, push to the NAS

**`tooling` branch only. Never push this to GitHub.** Like `TOOLING.md`, it names the fetch host,
the NAS hosts and their paths.

How to do by hand everything the timers normally do, plus the second NAS hop that feeds
`cosmo_web`. `TOOLING.md` has the details behind each step.

## Where things go

1. **mattellite → ana-nas.** Every scrape script ends by rsyncing its new files to
   `ana-nas:/volume1/rslsync/sync/Instagram-archive/archives/` (10.20.28.200).
   <https://instaarchive.ergosteur.com> reads straight from there. **There is no separate step
   for this hop.**
2. **ana-nas → DH2300.** The `archiver-instagram` container on `DH2300-D7CA` pulls that archive,
   makes thumbnails, remuxes video, normalizes into `cosmo_web`'s data and translates captions.
   It runs weekly (Wednesday 04:00). Step 6 below runs it by hand.

## 1. Pre-flight (mattellite)

```sh
ssh matt@mattellite
~/gdl/check-session.py                                   # want: "session ok ... expires <date>"
curl -s http://localhost:9222/json/version | head -2     # Chrome DevTools port is up
~/gdl/gdl-upgrade.sh                                     # gallery-dl + yt-dlp to latest
```

If the session check fails, log in to Instagram in the VNC Chrome (display `:1`), **reload the
tab** (a dead session can still look logged in) and re-check.

## 2. Stories — first; they expire in 24 h

```sh
~/gdl/run-stories.sh
```

About 2 minutes. `rsync ... (code 23)` / `failed to set permissions` is expected: the script
says "files landed" when that is the only problem. `No results` for a profile just means it has
no active story.

## 3. Posts and reels

**While gallery-dl's listing is broken** ([gh#9829](https://github.com/mikf/gallery-dl/issues/9829),
from 2026-10-22), use the browser scraper, one profile at a time:

```sh
for p in heejin kimxxlip; do ~/gdl/posts-sync.sh $p; done
for p in withaseul cher_ryppo zindoriyam; do ~/gdl/posts-sync.sh $p; done
~/gdl/posts-sync.sh official_artms
```

- `posts-sync.sh` scrolls `/<user>/` in the signed-in Chrome, stops once it reaches posts the
  archive already holds, and fetches only the new ones as single posts. Each profile takes a few
  minutes, mostly the 12–20 s gap between fetches.
- It prints `no new posts for <user>` when a profile is caught up.
- Logs: `~/gdl/logs/posts-<user>-*.log`.
- Reels also show up in the main grid. `~/gdl/run-reels-all.sh` sweeps the separate Reels tab
  too (~15 min) and is rarely needed.

**Once a gallery-dl release fixes the listing**, one command covers posts, reels, stories and
highlights for all six profiles:

```sh
~/gdl/run-profiles.sh        # ~5 min
```

For long runs, detach:
`setsid nohup ~/gdl/run-profiles.sh >/dev/null 2>&1 &`, then
`tail -f ~/gdl/logs/$(ls -1t ~/gdl/logs | head -1)`.

**Stop at once** on a `429` from `scontent-*.cdninstagram.com` or a `400 Bad Request` on
`/api/v1/feed/reels_media/`. See "When a run fails" in `TOOLING.md`.

## 4. Highlights

These only come through `run-profiles.sh`. Expect them to fail with the same
`Requested user could not be found` until gh#9829 is fixed in a release, so skip them until then.

## 5. Check it landed (any machine)

```sh
for u in heejin kimxxlip withaseul cher_ryppo zindoriyam official_artms; do
  n=$(curl -s "https://instaarchive.ergosteur.com/api/archives/$u/files" | python3 -c 'import json,sys; d=json.load(sys.stdin); print(len(d if isinstance(d,list) else d["files"]))')
  printf '%-16s %s\n' "$u" "$n"
done
```

Or browse <https://instaarchive.ergosteur.com>. New files appear without a redeploy.

## 6. Push to `cosmo_web` (the DH2300 hop)

This runs the same command the container's cron line runs (from `docker/archiver-instagram-entrypoint.sh`
in Cosmo-Live-Downloads):

```sh
ssh matt@DH2300-D7CA
docker exec cosmo-web-archiver-instagram-1 sh -c \
  'cd /work && . /app/instagram-archiver.env && python3 scripts/instagram_archive_run.py'
docker logs --tail 50 cosmo-web-archiver-instagram-1
```

It pings that container's Healthchecks check, same as a scheduled run. The pipeline itself
(mirror, thumbnail, remux, normalize, collaborators, translate) is described in the docstring of
`scripts/instagram_archive_run.py` in Cosmo-Live-Downloads.