Manual run — scrape, archive, push to the NAS

tooling branch only. Never push this to GitHub. Like TOOLING.md, it names the fetch host, the NAS hosts and their paths.

How to do by hand everything the timers normally do, plus the second NAS hop that feeds cosmo_web. TOOLING.md has the details behind each step.

Where things go

mattellite → ana-nas. Every scrape script ends by rsyncing its new files to ana-nas:/volume1/rslsync/sync/Instagram-archive/archives/ (10.20.28.200). https://instaarchive.ergosteur.com reads straight from there. There is no separate step for this hop.
ana-nas → DH2300. The archiver-instagram container on DH2300-D7CA pulls that archive, makes thumbnails, remuxes video, normalizes into cosmo_web's data and translates captions. It runs weekly (Wednesday 04:00). Step 6 below runs it by hand.

1. Pre-flight (mattellite)

ssh matt@mattellite
~/gdl/check-session.py                                   # want: "session ok ... expires <date>"
curl -s http://localhost:9222/json/version | head -2     # Chrome DevTools port is up
~/gdl/gdl-upgrade.sh                                     # gallery-dl + yt-dlp to latest

If the session check fails, log in to Instagram in the VNC Chrome (display :1), reload the tab (a dead session can still look logged in) and re-check.

2. Stories — first; they expire in 24 h

~/gdl/run-stories.sh

About 2 minutes. rsync ... (code 23) / failed to set permissions is expected: the script says "files landed" when that is the only problem. No results for a profile just means it has no active story.

3. Posts and reels

While gallery-dl's listing is broken (gh#9829, from 2026-10-22), use the browser scraper, one profile at a time:

for p in heejin kimxxlip; do ~/gdl/posts-sync.sh $p; done
for p in withaseul cher_ryppo zindoriyam; do ~/gdl/posts-sync.sh $p; done
~/gdl/posts-sync.sh official_artms

posts-sync.sh scrolls /<user>/ in the signed-in Chrome, stops once it reaches posts the archive already holds, and fetches only the new ones as single posts. Each profile takes a few minutes, mostly the 12–20 s gap between fetches.
It prints no new posts for <user> when a profile is caught up.
Logs: ~/gdl/logs/posts-<user>-*.log.
Reels also show up in the main grid. ~/gdl/run-reels-all.sh sweeps the separate Reels tab too (~15 min) and is rarely needed.

Once a gallery-dl release fixes the listing, one command covers posts, reels, stories and highlights for all six profiles:

~/gdl/run-profiles.sh        # ~5 min

For long runs, detach: setsid nohup ~/gdl/run-profiles.sh >/dev/null 2>&1 &, then tail -f ~/gdl/logs/$(ls -1t ~/gdl/logs | head -1).

Stop at once on a 429 from scontent-*.cdninstagram.com or a 400 Bad Request on /api/v1/feed/reels_media/. See "When a run fails" in TOOLING.md.

4. Highlights

These only come through run-profiles.sh. Expect them to fail with the same Requested user could not be found until gh#9829 is fixed in a release, so skip them until then.

5. Check it landed (any machine)

for u in heejin kimxxlip withaseul cher_ryppo zindoriyam official_artms; do
  n=$(curl -s "https://instaarchive.ergosteur.com/api/archives/$u/files" | python3 -c 'import json,sys; d=json.load(sys.stdin); print(len(d if isinstance(d,list) else d["files"]))')
  printf '%-16s %s\n' "$u" "$n"
done

Or browse https://instaarchive.ergosteur.com. New files appear without a redeploy.

6. Push to `cosmo_web` (the DH2300 hop)

This runs the same command the container's cron line runs (from docker/archiver-instagram-entrypoint.sh in Cosmo-Live-Downloads):

ssh matt@DH2300-D7CA
docker exec cosmo-web-archiver-instagram-1 sh -c \
  'cd /work && . /app/instagram-archiver.env && python3 scripts/instagram_archive_run.py'
docker logs --tail 50 cosmo-web-archiver-instagram-1

It pings that container's Healthchecks check, same as a scheduled run. The pipeline itself (mirror, thumbnail, remux, normalize, collaborators, translate) is described in the docstring of scripts/instagram_archive_run.py in Cosmo-Live-Downloads.

4.2 KiB Raw Blame History Unescape Escape