Manual run — scrape, archive, push to the NAS

tooling branch only. Never push this to GitHub. Like TOOLING.md, it names the fetch host, the NAS hosts and their paths.

How to do by hand everything the timers normally do, plus the second NAS hop that feeds cosmo_web. TOOLING.md has the details behind each step.

Where things go

mattellite → ana-nas. Every scrape script ends by rsyncing its new files to ana-nas:/volume1/rslsync/sync/Instagram-archive/archives/ (10.20.28.200). https://instaarchive.ergosteur.com reads straight from there. There is no separate step for this hop.
ana-nas → DH2300. The archiver-instagram container on DH2300-D7CA pulls that archive, makes thumbnails, remuxes video, normalizes into cosmo_web's data and translates captions. It runs weekly (Wednesday 04:00). Step 6 below runs it by hand.

1. Pre-flight (mattellite)

ssh matt@mattellite
~/gdl/check-session.py                                   # want: "session ok ... expires <date>"
curl -s http://localhost:9222/json/version | head -2     # Chrome DevTools port is up
~/gdl/gdl-upgrade.sh                                     # gallery-dl + yt-dlp to latest

If the session check fails, log in to Instagram in the VNC Chrome (display :1), reload the tab (a dead session can still look logged in) and re-check.

2. Stories — first; they expire in 24 h

~/gdl/run-stories.sh

About 2 minutes. rsync ... (code 23) / failed to set permissions is expected: the script says "files landed" when that is the only problem. No results for a profile just means it has no active story.

3. Posts and reels

While gallery-dl's listing is broken (gh#9829, from 2026-10-22), use the browser scraper, one profile at a time:

for p in heejin kimxxlip; do ~/gdl/posts-sync.sh $p; done
for p in withaseul cher_ryppo zindoriyam; do ~/gdl/posts-sync.sh $p; done
~/gdl/posts-sync.sh official_artms

posts-sync.sh scrolls /<user>/ in the signed-in Chrome, stops once it reaches posts the archive already holds, and fetches only the new ones as single posts. Each profile takes a few minutes, mostly the 12–20 s gap between fetches.
It prints no new posts for <user> when a profile is caught up.
Logs: ~/gdl/logs/posts-<user>-*.log.
Then sweep the Reels tab too: ~/gdl/run-reels-all.sh (~15 min). Most reels never appear in the main grid. On 2026-10-23 only 3 of heejin's 114 reels did, so posts-sync.sh alone misses them. reels-sync.sh <user> does a single profile.

Or the whole thing in one command, the same run the weekly timer does: gallery-dl for stories and highlights, then the browser for every profile's posts grid and Reels tab, each stopping once it reaches archived content:

~/gdl/run-profiles.sh

Once a gallery-dl release fixes the listing, GDL_LISTING=api ~/gdl/run-profiles.sh does posts and reels through gallery-dl again (~5 min).

For long runs, detach: setsid nohup ~/gdl/run-profiles.sh >/dev/null 2>&1 &, then tail -f ~/gdl/logs/$(ls -1t ~/gdl/logs | head -1).

Stop at once on a 429 from scontent-*.cdninstagram.com or a 400 Bad Request on /api/v1/feed/reels_media/. See "When a run fails" in TOOLING.md.

4. Highlights

These only come through run-profiles.sh. Expect them to fail with the same Requested user could not be found until gh#9829 is fixed in a release, so skip them until then.

5. Check it landed (any machine)

for u in heejin kimxxlip withaseul cher_ryppo zindoriyam official_artms; do
  n=$(curl -s "https://instaarchive.ergosteur.com/api/archives/$u/files" | python3 -c 'import json,sys; d=json.load(sys.stdin); print(len(d if isinstance(d,list) else d["files"]))')
  printf '%-16s %s\n' "$u" "$n"
done

Or browse https://instaarchive.ergosteur.com. New files appear without a redeploy.

6. Push to `cosmo_web` (the DH2300 hop)

This runs the same command the container's cron line runs (from docker/archiver-instagram-entrypoint.sh in Cosmo-Live-Downloads):

ssh matt@DH2300-D7CA
docker exec cosmo-web-archiver-instagram-1 sh -c \
  'cd /work && . /app/instagram-archiver.env && python3 scripts/instagram_archive_run.py'
docker logs --tail 50 cosmo-web-archiver-instagram-1

It pings that container's Healthchecks check, same as a scheduled run. The pipeline itself (mirror, thumbnail, remux, normalize, collaborators, translate) is described in the docstring of scripts/instagram_archive_run.py in Cosmo-Live-Downloads.

4.5 KiB Raw Permalink Blame History Unescape Escape