110 lines
4.5 KiB
Markdown
110 lines
4.5 KiB
Markdown
# Manual run — scrape, archive, push to the NAS
|
||
|
||
**`tooling` branch only. Never push this to GitHub.** Like `TOOLING.md`, it names the fetch host,
|
||
the NAS hosts and their paths.
|
||
|
||
How to do by hand everything the timers normally do, plus the second NAS hop that feeds
|
||
`cosmo_web`. `TOOLING.md` has the details behind each step.
|
||
|
||
## Where things go
|
||
|
||
1. **mattellite → ana-nas.** Every scrape script ends by rsyncing its new files to
|
||
`ana-nas:/volume1/rslsync/sync/Instagram-archive/archives/` (10.20.28.200).
|
||
<https://instaarchive.ergosteur.com> reads straight from there. **There is no separate step
|
||
for this hop.**
|
||
2. **ana-nas → DH2300.** The `archiver-instagram` container on `DH2300-D7CA` pulls that archive,
|
||
makes thumbnails, remuxes video, normalizes into `cosmo_web`'s data and translates captions.
|
||
It runs weekly (Wednesday 04:00). Step 6 below runs it by hand.
|
||
|
||
## 1. Pre-flight (mattellite)
|
||
|
||
```sh
|
||
ssh matt@mattellite
|
||
~/gdl/check-session.py # want: "session ok ... expires <date>"
|
||
curl -s http://localhost:9222/json/version | head -2 # Chrome DevTools port is up
|
||
~/gdl/gdl-upgrade.sh # gallery-dl + yt-dlp to latest
|
||
```
|
||
|
||
If the session check fails, log in to Instagram in the VNC Chrome (display `:1`), **reload the
|
||
tab** (a dead session can still look logged in) and re-check.
|
||
|
||
## 2. Stories — first; they expire in 24 h
|
||
|
||
```sh
|
||
~/gdl/run-stories.sh
|
||
```
|
||
|
||
About 2 minutes. `rsync ... (code 23)` / `failed to set permissions` is expected: the script
|
||
says "files landed" when that is the only problem. `No results` for a profile just means it has
|
||
no active story.
|
||
|
||
## 3. Posts and reels
|
||
|
||
**While gallery-dl's listing is broken** ([gh#9829](https://github.com/mikf/gallery-dl/issues/9829),
|
||
from 2026-10-22), use the browser scraper, one profile at a time:
|
||
|
||
```sh
|
||
for p in heejin kimxxlip; do ~/gdl/posts-sync.sh $p; done
|
||
for p in withaseul cher_ryppo zindoriyam; do ~/gdl/posts-sync.sh $p; done
|
||
~/gdl/posts-sync.sh official_artms
|
||
```
|
||
|
||
- `posts-sync.sh` scrolls `/<user>/` in the signed-in Chrome, stops once it reaches posts the
|
||
archive already holds, and fetches only the new ones as single posts. Each profile takes a few
|
||
minutes, mostly the 12–20 s gap between fetches.
|
||
- It prints `no new posts for <user>` when a profile is caught up.
|
||
- Logs: `~/gdl/logs/posts-<user>-*.log`.
|
||
- **Then sweep the Reels tab too:** `~/gdl/run-reels-all.sh` (~15 min). Most reels never
|
||
appear in the main grid. On 2026-10-23 only 3 of heejin's 114 reels did, so `posts-sync.sh`
|
||
alone misses them. `reels-sync.sh <user>` does a single profile.
|
||
|
||
**Or the whole thing in one command**, the same run the weekly timer does: gallery-dl for
|
||
stories and highlights, then the browser for every profile's posts grid and Reels tab, each
|
||
stopping once it reaches archived content:
|
||
|
||
```sh
|
||
~/gdl/run-profiles.sh
|
||
```
|
||
|
||
Once a gallery-dl release fixes the listing, `GDL_LISTING=api ~/gdl/run-profiles.sh` does posts
|
||
and reels through gallery-dl again (~5 min).
|
||
|
||
For long runs, detach:
|
||
`setsid nohup ~/gdl/run-profiles.sh >/dev/null 2>&1 &`, then
|
||
`tail -f ~/gdl/logs/$(ls -1t ~/gdl/logs | head -1)`.
|
||
|
||
**Stop at once** on a `429` from `scontent-*.cdninstagram.com` or a `400 Bad Request` on
|
||
`/api/v1/feed/reels_media/`. See "When a run fails" in `TOOLING.md`.
|
||
|
||
## 4. Highlights
|
||
|
||
These only come through `run-profiles.sh`. Expect them to fail with the same
|
||
`Requested user could not be found` until gh#9829 is fixed in a release, so skip them until then.
|
||
|
||
## 5. Check it landed (any machine)
|
||
|
||
```sh
|
||
for u in heejin kimxxlip withaseul cher_ryppo zindoriyam official_artms; do
|
||
n=$(curl -s "https://instaarchive.ergosteur.com/api/archives/$u/files" | python3 -c 'import json,sys; d=json.load(sys.stdin); print(len(d if isinstance(d,list) else d["files"]))')
|
||
printf '%-16s %s\n' "$u" "$n"
|
||
done
|
||
```
|
||
|
||
Or browse <https://instaarchive.ergosteur.com>. New files appear without a redeploy.
|
||
|
||
## 6. Push to `cosmo_web` (the DH2300 hop)
|
||
|
||
This runs the same command the container's cron line runs (from `docker/archiver-instagram-entrypoint.sh`
|
||
in Cosmo-Live-Downloads):
|
||
|
||
```sh
|
||
ssh matt@DH2300-D7CA
|
||
docker exec cosmo-web-archiver-instagram-1 sh -c \
|
||
'cd /work && . /app/instagram-archiver.env && python3 scripts/instagram_archive_run.py'
|
||
docker logs --tail 50 cosmo-web-archiver-instagram-1
|
||
```
|
||
|
||
It pings that container's Healthchecks check, same as a scheduled run. The pipeline itself
|
||
(mirror, thumbnail, remux, normalize, collaborators, translate) is described in the docstring of
|
||
`scripts/instagram_archive_run.py` in Cosmo-Live-Downloads.
|